ScoreFlow: Mastering LLM Agent Workflows via Score-based Preference Optimization

1 Introduction

• Automated workflow methods face restricted search spaces from predefined structures, and discrete optimization can limit exploration, adaptability, and scalability.Prior methods may optimize one workflow for an entire task set, limiting adaptation to diverse problems.Source: §1 · p. 1
• ScoreFlow generates a code-represented workflow for each task and optimizes its generator using evaluation-score feedback.The framework is designed to reduce human intervention in multi-agent workflow generation and optimization.Source: §1 · pp. 1–2
• Loss-gradient optimization is described as more flexible and scalable than prior discrete methods, while open-source generator models reduce workflow-generation API costs.The contributions also report robustness, scalability, and cost-efficiency across different models.Source: §1 · p. 2
• Score-DPO incorporates quantitative evaluation scores into preference optimization rather than relying only on preference pairs.Evaluation-score variance and inaccuracies can reduce pair reliability, slowing convergence and limiting performance within a fixed number of iterations.Source: §1 · p. 2
• 8.2%: ScoreFlow outperforms baseline methods across six benchmarks spanning question answering, coding, and mathematical reasoning.The framework also enables smaller models to outperform larger ones with greater cost efficiency.Source: §1 · p. 2

2.1 Agentic Workflow Optimization

• Workflow-optimization methods target prompts, hyperparameters, or workflow structure, but representation limits and discrete search can constrain flexibility and exploration.ADAS uses code representation but faces search complexity; AFlow uses Monte Carlo Tree Search but may converge prematurely.Source: §2.1 · p. 2

2.2 Learning from Preferences for Language Models

• PPO trains a reward model on preferred and rejected responses, then optimizes the policy while applying a soft KL-divergence constraint.The stated constraint is intended to prevent degeneration.Source: §2.2 · pp. 2–3
• DPO directly optimizes a policy from preference data, avoiding explicit reward models or active policy sampling.The pipeline generates multiple workflows per problem, evaluates them, forms preference data from scores, and fine-tunes the generator with Score-DPO.Source: §2.2 · p. 3, Figure 1 · p. 3
• Score-DPO integrates associated evaluation scores into training and reports improved performance over standard DPO while retaining its efficiency and stability.This setting includes scores for each sample rather than only chosen and rejected pairs.Source: §2.2 · p. 3

3.1 Background

• Workflows pair an agent set with an executable function; code supports sequences, loops, and conditionals, while a task-conditioned open-source generator can customize operator prompts.Source: §3.1 · pp. 3–4

3.2 ScoreFlow Overview

• At each iteration, ScoreFlow generates and evaluates multiple workflows per task, fine-tunes the generator with Score-DPO, and repeats until convergence or the iteration limit.Source: §3.2 · p. 4

3.3 Quantitative Labeling of Preference Workflows

• Each task’s workflows are executed and scored using an independent executor; higher-scoring and lower-scoring workflows form preference pairs with their scores.Experiments derive scores from average F1 or win rate using an independent executor LLM.Source: §3.3 · pp. 4–5

3.4 Optimization via Score-DPO

• Score-DPO upweights preference pairs with larger score gaps and incorporates evaluation scores into ranking, giving more deterministic evaluations greater influence on the loss.The sampling probability increases with the score difference; the ranking objective also adjusts implicit rewards using scores.Source: §3.4 · p. 5

3.5 Analysis of Score-DPO

• When |r_z| ≤ 1, Score-DPO’s per-sample influence increases with score; this sufficient condition held approximately 91.1% of the time before convergence.The theorem states the monotonicity condition, and the text reports the approximate frequency observed during optimization.Source: §3.5 · pp. 5–6

4.1 Experimental Setup

• Experiments cover six math, question-answering, and coding datasets, with static and automated workflow baselines and a default Llama-3.1-8B-Instruct generator.The default executor is GPT-4o-mini; solve rates are averaged over three evaluations, with F1 used for DROP and HotpotQA. Baselines include direct LLM invocation, prompting and debate workflows, ADAS, and AFlow.Source: §4.1 · p. 6

4.2 Results and Analysis

• 85.3% average solve rate, surpassing baseline methods by 8.2% across all six benchmarks.The automated baselines use GPT-4o-mini as their workflow generator, while ScoreFlow uses an 8B generator.Source: §4.2 · p. 6, Table 1 · p. 7
• Score-DPO achieves the best performance among the SFT, PPO, and raw DPO alternatives within the ScoreFlow pipeline.The authors note that SFT uses preferred responses only, while PPO's long-token-sequence optimization can dilute gradient signals and destabilize performance [30] [8].Source: §4.2 · p. 7, Table 2 · p. 7
• On the combined GSM8K, MBPP, and DROP task, ScoreFlow's test win-rate advantage over AFlow is more pronounced than on individual datasets.The combined task reports 88.1% performance for ScoreFlow.Source: §4.2 · p. 7, Figure 3 · p. 8
• Adaptive workflows match task complexity: program and review operators help on a complex problem, while an overly complex workflow can hinder a simple one.The two GSM8K problems are assigned distinct workflows, and the correct answer is achievable only with the corresponding workflow.Source: §4.2 · pp. 7–8
• Across generator and executor combinations, ScoreFlow outperforms baselines; GPT-4o and DeepSeek-coder as generators achieve the best performance.With smaller GPT-4o-mini or DeepSeek-V3 models, ScoreFlow also outperforms GPT-4o Chain-of-Thought outputs; workflow optimization brings DeepSeek-V3 to performance comparable to GPT-4o and DeepSeek-coder.Source: §4.2 · p. 8, Table 3 · p. 8
• ScoreFlow's optimization process consistently costs less than AFlow's while performing better on HumanEval.Figure 4 compares API costs during inference and optimization; its inference panel relates testing-set cost to Pass@1.Source: §4.2 · p. 9, Figure 4 · p. 9
• Test solve rates rise and then converge during optimization, with rapid convergence observed as early as iteration two.Figure 5 presents solve rate over the iteration process.Source: §4.2 · p. 9, Figure 5 · p. 9

5 Conclusion

• Score-DPO incorporates evaluation scores into preference optimization to reduce inaccuracies and variances in collected data pairs.The conclusion links this approach to improved overall performance.Source: §5 · p. 9
• 8.2% average improvement over baseline methods across six benchmarks spanning question answering, coding, and mathematical reasoning.The framework achieves an average improvement of 8.2% over baseline methods.Source: §5 · p. 9
• Smaller models can outperform larger models while incurring lower API costs.Source: §5 · p. 10

A.1 Proofs

• F(x) is strictly monotonically increasing for x > 0 when a ≤ 1/x.Source: §A.1 · p. 14
• Under condition −(1 − f(sz))^-1 ≤ rz ≤ f^-1(sz), Iw(z) increases and Il(z) decreases with sz.Source: §A.1 · p. 14
• The proof uses these opposing monotonic trends to conclude that I(z) is strictly monotonically increasing with sz.Source: §A.1 · p. 14

A.2.1 Generator Prompt

• The prompt requests an optimized workflow graph based on a template, permitted operators, and the given task.Source: §A.2.1 · p. 15
• Prompt instructions prohibit embedding task-specific information in the graph and allow loops and conditional flow to build more complex graphs.The prompt also cautions against runtime failures and disallows creating or importing unprovided operators.Source: §A.2.1 · p. 15
• The appendix lists separate single-agent workflow templates for question answering, math, and coding tasks.The template imposes no prior knowledge of which structure to build or operator to choose.Source: §A.2.1 · pp. 15–16, §A.3.1 · p. 19

A.2.2 Operator Utilized

• Math workflows use Custom, Programmer, Ensemble, and Reviewer operators to generate, execute, select, and refine solutions.Source: §A.2.2 · p. 17
• Question-answering workflows use Custom, AnswerGenerate, Ensemble, and Reviewer operators, while coding workflows use CustomCodeGenerate, Ensemble, and Test.Source: §A.2.2 · p. 17
• An Answer Extractor Agent follows the final response in each workflow to remove redundancy and make evaluation concise and precise.Source: §A.2.2 · p. 17
• The operator descriptions specify each operator's function, input format, and how its output can feed a subsequent operator or become the final output.Source: §A.2.2 · p. 17

• Algorithm 1 contains the detailed procedure for ScoreFlow.Source: §A.2.3 · p. 18

Algorithm 1 ScoreFlow

• ScoreFlow generates k workflows per task, filters them by a quality condition, scores them with an executor LLM, and uses score-ranked preference pairs to update the generator with Score-DPO.The algorithm repeats generator updates for S samples and M iterations; the quality condition guards against runtime errors and execution-time-limit violations.Source: Algorithm 1 ScoreFlow · p. 18

A.3.1 Sampled workflows

• Sampled workflows combine task-specific operators such as repeated solution generation, ensembling, programming, and testing.Source: §A.3.1 · pp. 20–21

A.3.2 Case study on d(x, y)

• Increasing α in d(x, y) = (x − y)^α improves performance by upweighting more deterministic pairs, but extreme α = 100 degrades performance.The authors attribute the decline to discarding many preference pairs, which removes information useful for generalization; DPO is the α = 0 case when ignoring f(x).Source: §A.3.2 · p. 22

A.3.3 Experimental Validation for Condition in Theorem 3.2

• The condition |r_z| ≤ 1 holds with probabilities of 99.8% on MATH, 82.2% on DROP, and 91.2% on MBPP, averaging 91.1%.Source: §A.3.3 · p. 22

A.3.4 Detailed Cost Data

• The appendix includes detailed cost-value tables for test-data optimization and inference processes.Source: Table 5 · p. 23, Table 6 · p. 24