PaperlayerView exact revision on arXiv

Source-linked AI summary

QualityFlow: An Agentic Workflow for Program Synthesis Controlled by LLM Quality Checks

Yaojie Hu, Qiang Zhou, Qihong Chen, Xiaopeng Li, Linbo Liu, Dejiao Zhang, Amit Kachroo, Talha Oz, Omer Tripp

arXiv:2501.17167v2cs.SEcs.AI

TL;DR

Program synthesis systems must generate programs that satisfy unit tests despite unreliable synthesized tests and potentially drifting debugging trajectories. QualityFlow coordinates generation, testing, self-debugging, and LLM-based quality checks that control acceptance, clarification, and reversion. It achieves state-of-the-art results across MBPP, MBPP-EvalPlus, HumanEval, and HumanEval-EvalPlus, while test-quality checking improves pass@1 by 0.8% on MBPP with a stronger LLM.

  • Problem

    Program synthesis workflows can assume visible-test conformity, rely on faulty synthesized tests, and follow self-debugging trajectories that stagnate or degrade.

    Source: §1 · pp. 1–2

  • Method

    QualityFlow uses cooperating agents for code generation, test design, self-debugging, and Quality Checker decisions based on Imagined Execution.

    Source: §1 · p. 2, §4.2 · p. 6

  • Results

    QualityFlow achieves state-of-the-art performance on MBPP, MBPP-EvalPlus, HumanEval, and HumanEval-EvalPlus, while test-quality checking improves MBPP pass@1 by 0.8% with Sonnet.

    Source: §1 · p. 2, §4.3 · p. 8

  • Takeaways & Limitations

    QualityFlow shows that accurate quality checks can retain correct programs, mitigate faulty synthesized tests, and control workflow deviation.

    Source: §5 · p. 8, §6 · p. 9

    The Test Quality Checker can reduce overall pass@1 when a weaker LLM such as Opus is used, and finite test suites cannot cover every edge case.

    Source: §6 · p. 9

Abstract

from arXiv · show

We introduce QualityFlow, a dynamic agentic workflow for program synthesis. Given the English description of a programming problem and a set of unit tests, the model's goal is to synthesize the correct program that solves the problem and passes the tests. QualityFlow includes large language model (LLM) agents resembling a software development team, including code generation, testing, and self-debugging. We propose the LLM Quality Checker, which explicitly "imagines" whether the synthesized programs' execution would conform to the unit tests. The Quality Checks dynamically control the workflow, including actions to submit the final answer, clarify the problem statement, and revert previous workflow steps. Our experiments show that the Quality Checker can precisely accept any correct program, mitigate faulty synthesized tests, and prevent potential workflow deviation. QualityFlow establishes the state-of-the-art results on four program synthesis benchmarks: MBPP, HumanEval, and stricter evaluations from MBPP-EvalPlus and HumanEval-EvalPlus.

1 Introduction

QualityFlow addresses three weaknesses in agentic program synthesis—visible-test conformity assumptions, faulty synthesized tests, and drifting self-debugging—using quality-controlled workflow decisions. It reports state-of-the-art performance across MBPP, MBPP-EvalPlus, HumanEval, and HumanEval-EvalPlus.

Motivation

  • Existing methods assume generated code conforms to visible tests, while synthesized tests can mislead debugging and repeated debugging can degrade or stagnate.These limitations motivate dynamic workflow control rather than static multi-agent execution.Source: §1 · pp. 1–2

Contribution

  • QualityFlow combines code generation, test design, self-debugging, and a Quality Checker that accepts, continues, clarifies, or reverts workflow trajectories.The Quality Checker uses Imagined Execution to compare predicted program outputs with unit-test assertions.Source: §1 · p. 2
  • Imagined Execution predicts program correctness by emulating execution and checking conformity with unit-test outputs.This quality check explicitly examines whether synthesized programs satisfy the tests.Source: §1 · p. 2

Results

  • QualityFlow establishes state-of-the-art program-synthesis performance on MBPP, MBPP-EvalPlus, HumanEval, and HumanEval-EvalPlus.The contributions explicitly identify all four benchmark suites.Source: §1 · p. 2

Contribution

  • Diversified Prompting runs varied prompts in parallel so the Quality Checker can select a correct candidate after generation.The method is presented as a post-hoc strategy for increasing the chance of producing an accepted solution.Source: §1 · p. 2

2 Related Work

QualityFlow builds on large language models, agentic collaboration, self-reflection, and software-engineering practices for program synthesis. Its workflow is organized around generated programs, tests, quality checks, and controlled downstream processing.

Code LLM

  • Large language models support program synthesis and related code tasks including bug fixing, testing, and fuzzing.The related work positions LLMs as the foundation for code-generation workflows.Source: §2 · p. 2

Agentic workflow

  • Agentic workflows coordinate multiple LLMs to collaborate on producing answers, extending zero-shot synthesis with team-like processing.QualityFlow applies this paradigm specifically to program synthesis.Source: §2 · p. 2

Self-reflection

  • Self-reflection improves LLM responses through feedback on their own outputs without requiring additional training data or reinforcement learning.The cited work contrasts this approach with resource-intensive fine-tuning.Source: §2 · pp. 2–3

QualityFlow

  • QualityFlow checks intermediate code and accepts it as final when the quality check passes; otherwise, later agents perform more processing.This control pattern is illustrated in Figure 2 and operationalized in Algorithm 1.Source: Figure 2 · p. 3, §2 · p. 3

3 Methods

QualityFlow starts with program generation, uses synthesized tests for iterative debugging, and applies quality checks to control acceptance, clarification, reversion, and prompt diversification. The workflow’s checks are designed to retain correct programs while filtering unreliable intermediate artifacts.

3.1 Program Generator

  • The Program Generator creates an initial program from the problem statement and visible unit tests.No additional contextual information is available at this initial stage.Source: §3.1 · p. 3

3.2 Test Designer

  • The Test Designer synthesizes common-case tests, which produce higher workflow pass@1 accuracy than corner-case tests.These tests guide subsequent self-debugging.Source: §3.2 · p. 3

3.3 Self-Debugger

  • The Self-Debugger iteratively revises programs using synthesized-test executions, error messages, and actual return values.Self-debugging uses chain-of-thought reflection and stops when tests pass, the epoch limit is reached, or the Quality Checker accepts the program.Source: §3.3 · p. 3

3.4 Quality Checker

  • Imagined Execution emulates program execution for each test and accepts a program only when all predicted outputs match the asserted outputs.The Quality Checker reaches 98% precision and 98% recall on MBPP.Source: §3.4 · pp. 3–4
  • Synthesized-test quality is a major bottleneck because 62% of LLM-generated tests are incorrect and can mislead self-debugging.The Test Quality Checker filters tests by comparing their proclaimed outputs with outputs inferred from the problem statement.Source: §3.4 · p. 4

3.5 Problem Clarifier

  • When self-debugging candidates are rejected, the Problem Clarifier re-examines the statement and trajectory context before requesting a different implementation.This gives the generator a second chance under a clarified interpretation.Source: §3.5 · p. 4
  • If clarification also fails the quality check, QualityFlow reverts to the original generated program to avoid cumulative workflow deviation.The reversion relies on the Quality Checker’s reported 98% recall after clarification.Source: §3.5 · p. 4

3.6 Diversified Prompting

  • Diversified Prompting uses varied parallel prompts to generate candidates and relies on accurate post-hoc quality checking to select among them.It improves overall workflow performance by 4% for MBPP.Source: §3.6 · p. 4, Table 1 · p. 5

4 Evaluation

QualityFlow evaluates a flexible agentic workflow for program synthesis across four benchmarks, using quality checks to control code generation, testing, self-debugging, and workflow recovery. The evaluation finds state-of-the-art performance, with Code Quality Checker and Imagined Execution improving program selection and Test Quality Checker providing conditional benefits.

4.1 RQ1. Can QualityFlow achieve a new state-of-the-art performance on Python code generation benchmarks?

  • QualityFlow’s pass@1 consistently increases across workflow steps as multiple LLM agents improve the initial generated program without additional training.The workflow outperforms single-attempt zero-shot synthesis across all LLM backbones and benchmarks.Source: §4.1 · p. 5
  • QualityFlow establishes new state-of-the-art pass@1 performance on MBPP, MBPP-EvalPlus, HumanEval, and HumanEval-EvalPlus.MBPP reaches 94.2% pass@1, HumanEval reaches 98.8%, and HumanEval-EvalPlus reaches 89.6% with Imagined Execution and 87.8% with Python Checker, versus prior SOTA of 86.6%.Source: Table 2 · p. 5, §4.1 · pp. 5–6

4.2 RQ2. Can the Quality Checker navigate the workflow and improve accuracy?

  • The Code Quality Checker uses Imagined Execution to emulate synthesized-program execution against unit-test outcomes and navigate workflow decisions.It can accept passing programs, reject uncertain or incorrect programs, and control continuation, restarting, reverting, or submission.Source: §4.2 · pp. 6–7
  • 94.2% versus 80.2% final pass@1 on MBPP shows that adding the Code Quality Checker substantially improves QualityFlow over its ablated workflow.Without the checker, every program proceeds through all steps, allowing incorrect synthesized tests and self-debugging to change correct programs into incorrect ones.Source: §4.2 · pp. 6–7, Figure 3 · p. 6
  • Imagined Execution is critical because a simple Yes/No critic is overly optimistic, has lower specificity and accuracy, and can reduce MBPP pass@1 from 81.0% to 78.8%.Using a Python interpreter to execute visible tests creates a perfect classifier and the best HumanEval workflow performance in the reported comparison.Source: §4.2 · p. 7, Table 5 · p. 7

4.3 RQ3. Can the Test Quality Checker validate LLM-generated tests to improve self-debugging and overall results?

  • The Test Quality Checker predicts synthesized-test correctness from the problem statement and filters tests that could mislead self-debugging.On MBPP with Sonnet, 62.25% of synthesized tests are incorrect, and TQC recalls 79.13% of them.Source: §4.3 · pp. 7–8
  • TQC improves overall pass@1 by 0.8% on MBPP and 0.61% on HumanEval with Sonnet, but can reduce performance with the weaker Opus model.The reported results indicate that test-quality checking is more challenging than code-quality checking.Source: §4.3 · p. 8, Table 7 · p. 8
  • TQC remains limited, with about 80% recall compared with 98% recall for the Code Quality Checker, leaving room for better test-quality checkers.The limitation is especially relevant because synthesized-test quality is a major source of self-debugging errors.Source: §4.3 · p. 8

5 Discussions and Conclusions

QualityFlow identifies three program-synthesis workflow limitations and addresses them with a Quality Checker using Imagined Execution. Its central perspective is that correct programs must be identified and retained as final answers, not merely generated somewhere in the workflow.

  • QualityFlow targets visible-test conformity assumptions, faulty synthesized tests, and self-debugging trajectory deviation.Source: §5 · p. 8
  • The Quality Checker uses Imagined Execution to examine programs and control workflow actions, including rejection, restart, and reset decisions.Source: §5 · p. 8
  • QualityFlow’s key perspective is that program synthesis requires identifying and retaining correct programs as final answers.Source: §5 · p. 8

6 Limitations

The evaluation uses Claude models and public benchmarks, but generalization beyond these datasets and models remains uncertain. Additional limitations include incomplete test coverage, imperfect detection, possible artifact bugs, and adverse effects from weaker test-quality checking.

6 Limitations

  • Results may not generalize to datasets beyond the public benchmarks used with Anthropic Claude models.Source: §6 · p. 9
  • The Test Quality Checker can sometimes reduce pass@1 with weaker LLMs such as Opus.Source: §6 · p. 9
  • QualityChecker requires clear unit tests, which may be unavailable for every file in large projects with limited test coverage.Source: §6 · p. 9
  • The experiments may contain tool or script bugs, mitigated through code review, validity checks, repeated experiments, and planned public release.Source: §6 · p. 9
  • Pass@1 uses finite test suites, which cannot perfectly cover all edge cases.Source: §6 · p. 9

6.1 Potential risks

  • QualityFlow can still generate incorrect programs because Code Quality Checker detection is not perfect.Source: §6.1 · p. 9

6.2 License of artifacts

  • The paper reports using publicly accessible benchmark artifacts under permissible licenses and AI assistants for grammar, spelling, drafting, and debugging.Source: §6.2 · p. 9, §6.4 · p. 9

A QualityFlow hyper-parameters

QualityFlow uses diversified prompting to create parallel workflow progressions, followed by repeated self-debugging and clarification. Its generation settings are deterministic except for mildly randomized test design, which produces up to 50 tests per program before post-processing.

B DeepSeek integration with QualityFlow

QualityFlow is also evaluated with DeepSeek and compared with prior workflow results. It outperforms LDB across all benchmarks and remains better than zero-shot synthesis in all tested settings, while Sonnet-based QualityFlow performs best.

C LDB reproduction with Sonnet LLM

QualityFlow outperforms the reproduced LDB baseline under the same Sonnet LLM, while removing either Clarifier or Revert consistently reduces pass@1.

C LDB reproduction with Sonnet LLM

  • 98.8% pass@1 on HumanEval matches QualityFlow’s reported result and exceeds reproduced LDB’s 95.1% under the same Sonnet LLM.The matched-backbone comparison attributes the advantage to workflow design rather than different LLMs.Source: C LDB reproduction with Sonnet LLM · p. 12

D Removing the Clarifier or Revert mechanism

  • Removing either Clarifier or Revert reduces pass@1 in all six ablation settings, with absolute decreases from 0.78% to 2.44%.The ablations cover standard and relaxed QualityFlow variants on HumanEval and standard QualityFlow on MBPP.Source: D Removing the Clarifier or Revert… · p. 13

E AgentCoder reproduction

The AgentCoder reproduction reports 91.8% accuracy under pass@6 on MBPP, which the authors say conflicts with the original paper’s claims.

E AgentCoder reproduction

  • 91.8% accuracy is reported for AgentCoder’s MBPP pass@6 evaluation, contrary to the original paper’s claims.The authors state that the evaluation appears to use pass@6 rather than pass@1.Source: E AgentCoder reproduction · p. 13

F Token usages

QualityFlow dynamically allocates inference-time computation: Diversified Prompting increases per-agent token use but can enable earlier workflow termination through quality checks.

F Token usages

  • QualityFlow allocates more inference-time compute to harder problems dynamically under Code Quality Checker control.The workflow can skip later steps when a quality check accepts an intermediate program.Source: F Token usages · p. 13, Table 11 · p. 14
  • Diversified Prompting increases token usage for individual agents while increasing the probability that quality checks accept a program and skip later steps.Without Diversified Prompting, token usage is lower, but performance is also lower.Source: F Token usages · p. 13, Table 12 · p. 14
  • Table 11 reports average token counts for each QualityFlow component and total tokens per problem, including savings from skipping subsequent steps after acceptance.Table 12 reports corresponding token counts without Diversified Prompting.Source: Table 11 · p. 14, Table 12 · p. 14

G Examples of Code Quality Checker input and output (Imagined Execution)

The Code Quality Checker uses Imagined Execution to predict unit-test outcomes, correctly handling several examples but also exhibiting reasoning failures and false acceptance.

G Examples of Code Quality Checker input and output (Imagined Execution)

  • The examples include correctly accepted and rejected cases alongside incorrectly accepted and rejected cases to illustrate Code Quality Checker behavior.The examples are drawn from MBPP and use function-and-unit-test input/output conversations.Source: G Examples of Code Quality Checker… · pp. 15, 16, 18
  • The checker correctly reasons that first_Digit(123) returns 1, and parsing recovers the answer even when the LLM omits the requested test-case tag.Correct Imagined Execution constitutes most checker runs because of its high precision.Source: G Examples of Code Quality Checker… · p. 15
  • For replace_char('polygon', 'y', 'l'), the reasoning predicts 'polgon', while the passage identifies the correct execution result as 'pollgon'.This illustrates an Imagined Execution mistake in finding the expected result.Source: G Examples of Code Quality Checker… · p. 16
  • The checker can incorrectly accept faulty code when Imagined Execution predicts π/2 although the program actually returns π for the tested complex-number input.The canonical test compares against 1.5707963267948966, exposing the mistaken imagined result.Source: G Examples of Code Quality Checker… · pp. 17–18
  • The checker correctly rejects code when execution produces B: Blue but the canonical test expects B: Black in the merged dictionary.The explicit mapping analysis supports the strength of Imagined Execution in predicting synthesized-code correctness.Source: G Examples of Code Quality Checker… · pp. 18–19

Related papers

Loading 2501.17167v2…