Optimizing LLM-Based Multi-Agent System with Textual Feedback: A Case Study on Software Development

• The study focuses on role-based LLM multi-agent systems that collaboratively solve complex software development tasks.Source: §1 · pp. 1–2
• Prior prompt-optimization methods had not conclusively demonstrated effectiveness in genuine role-based multi-agent systems, and primarily targeted traditional NLP benchmarks.Source: §1 · pp. 1–2
• The proposed pipeline locates underperforming agents using textual feedback and failure explanations, then optimizes their system prompts.Source: §1 · p. 2
• The study evaluates software-development optimization across diverse dimensions and compares online versus offline, individual versus group, and one-pass versus multi-pass settings.Source: §1 · p. 2
• The authors report that their pipeline effectively optimizes the system across the studied evaluation dimensions and settings, with online and group optimization more effective than individual and offline optimization.Source: §1 · p. 2

2.1 LLM-Based Agents

• LLM-based agents have been combined into role-based systems in which distinct agent roles collaborate on software development and other complex tasks.Source: §2.1 · pp. 2–3

2.2 Prompt Optimization

• Automatic prompt optimization includes resampling-based methods and reflection-based methods that use feedback or historical information to refine prompts.Source: §2.2 · p. 3
• This work uses an LLM optimizer to refine prompts of underperforming agents and places its optimization step in the reflection-based category.Source: §2.2 · p. 3

2.3 Agent Optimization

• Existing systems such as DSPy and TextGrad had not experimentally established effectiveness in role-based multi-agent settings, while graph-optimization approaches assume decentralized agents and consensus-style outputs.Source: §2.3 · p. 3
• Unlike prior work focused on traditional NLP tasks, this paper studies optimization for complex, open-ended software development tasks.Source: §2.3 · p. 3

3 Software Developemt Task

• Software development requires complete solution code from detailed requirements spanning applications such as board games and social networking.Source: §3 · p. 3

3.1 Why Software Development?

• Software development is a suitable test bed because it requires cooperation among agents with diverse skills and lacks standard ground-truth solutions or evaluation metrics.Source: §3.1 · p. 4
• The absence of fixed ground truth provides flexibility to evaluate optimization across user-defined dimensions, while group optimization in this setting had not previously been addressed.Source: §3.1 · p. 4

3.2 Optimization Dimensions

• The five evaluation dimensions are functionality, robustness, test case coverage, documentation, and code style violation.Source: §3.2 · p. 4

4.1 Problem Formulation

• The system is modeled as a directed graph whose nodes are LLM agents, whose edges represent message passing, and whose role assignments are reflected in agent prompts.Source: §4.1 · p. 4
• A critic evaluates the system’s final response with a scalar utility and natural-language feedback across five software-development dimensions, and the optimized prompt group is evaluated on a test set.Source: §4.1 · p. 5

4.2 Optimization Pipeline

• The two-step pipeline first uses a locator to identify underperforming agents and explain their failures, then uses an optimizer to revise those agents’ system prompts.Source: §4.2 · p. 5

4.2.1 Locating

• The locator uses the task, global feedback, role descriptions, and communication trajectories to produce failing-agent identities and fine-grained failure explanations.Source: §4.2.1 · p. 5

4.2.2 Optimizing

• The optimizer uses each identified agent’s prompt, messages, output, and failure explanation to modify its prompt toward mitigating the diagnosed failure.Source: §4.2.2 · pp. 5–6

4.3.1 Online against Offline Optimization

• Online optimization collects feedback using prompts current at each step, whereas offline optimization collects feedback beforehand using the default prompts.Source: §4.3 · p. 6, §4.3.1 · p. 6

4.3.2 Individual against Group Optimization

• Individual optimization updates one sampled underperforming agent per step, while group optimization updates all identified underperforming agents at each step.Source: §4.3.2 · p. 6
• One-pass prompting jointly optimizes all identified agents in one inference call, whereas multi-pass prompting uses separate calls; one-pass is more efficient but may include irrelevant agent information.Source: §4.3.2 · p. 6

5.1 Dataset

• SRDD contains 1,200 software task prompts across five categories, split randomly into train, development, and test sets at 6:2:2.Source: §5.1 · p. 6

5.2 Critic Mechanism Implementation

• Critics evaluate software solutions along five dimensions using scalar scores and natural-language feedback generated by rules, external tools, or LLM judges.Source: §5.2 · p. 7
• Functionality, robustness, and test coverage receive GPT-4 scores from 0 to 10, while documentation and style use comment counts and pycodestyle violations.Source: §5.2 · p. 7

5.3 Experiment Setting

• Experiments use gpt-4-0613 at temperature 0.1, sample five training tasks per optimization step, and evaluate 100 test tasks unless otherwise stated.Source: §5.3 · p. 7

5.4 Baselines

• Baselines comprise default prompts, one-shot demonstrations shared across agents, and direct optimization of all prompts from textual feedback.Source: §5.4 · p. 7

5.5 Main Results

• The proposed pipeline outperforms both baselines across all reported settings and dimensions except offline individual functionality, where it trails the unoptimized system by 0.1.Source: §5.5 · p. 8
• Online group optimization with one-pass prompting is consistently among the top two settings across evaluation dimensions and is identified as the most effective strategy.Source: §5.5 · p. 8
• Group optimization usually outperforms individual optimization, while offline optimization is worse than online optimization on only two of five dimensions.Source: §5.5 · p. 8
• One-pass and multi-pass group prompting show no consistent performance difference, making one-pass prompting a more efficient choice under this setup.Source: §5.5 · p. 8

5.6.1 Optimization Curve Analysis

• Training and development curves remain consistent across dimensions, indicating no observed overfitting under online group one-pass optimization.Source: §5.6.1 · p. 9
• Model-based critics produce more oscillatory training curves than rule-based critics, while training curves are less stable than development curves.Source: §5.6.1 · p. 9
• Online optimization yields substantially more stable code-style training curves than offline optimization under both one-pass and multi-pass prompting.Source: §5.6.1 · p. 9, Figure 3 · p. 9

5.6.2 Starting from "Empty" Prompts

• Starting from a minimally contextualized empty prompt produces performance on par with or only slightly below optimization from an informative default prompt.Source: §5.6.2 · p. 9

5.6.3 Case Study

• Optimized prompts add dimension-specific guidance, such as implementing all requested functionality, reviewing missing functionality, and enforcing PEP8 style.Source: §5.6.3 · p. 9, Table 2 · p. 10
• Optimized prompts sometimes include instance-specific training content, motivating future work on improving prompt generalization.Source: §5.6.3 · p. 9

• The study combines two-step prompt optimization with an investigation of optimization settings to improve role-based multi-agent software development.The framework uses natural-language feedback and examines group optimization behavior under different settings.Source: §6 · p. 10

• A solver proposes code, two reviewers provide feedback, and the solver refines the solution in a single review iteration.The case study uses two reviewer agents and one refinement round.Source: A Role-based Multi-Agent System · p. 16
• A recruiter selects concrete solver and reviewer roles from a predefined expert pool based on the software task description.The role-description pool is adapted from ChatDev.Source: A Role-based Multi-Agent System · p. 16
• The optimization targets system prompts for all agents in the vertical decision-making structure.The solver prompt is constrained to the programmer role because it writes the solution code.Source: A Role-based Multi-Agent System · p. 16

• Evaluation prompts generate scalar scores and feedback for functionality, robustness, and test-case coverage, plus feedback for documentation.The evaluation dimension name and definition, task description, and software solution code are key prompt components.Source: B Prompt for Model-Based Evaluation · pp. 16–17
• Agent prompts include the agent role, software task description, and chat history with other teammates.Source: B Prompt for Model-Based Evaluation · p. 17
• The reviewer agent reviews solver-written solution code and provides feedback given the software task description.Source: Figure 6 · p. 17
• The evaluation dimensions used in the study are defined concretely in Table 3.Source: Table 3 · p. 17, B Prompt for Model-Based Evaluation · p. 17

• Training and development curves became stable or declined when optimization increased from 5 to 10 steps, indicating that more steps were unnecessary in this case study.The test used functionality and code-style-violation dimensions under online group optimization with one-pass prompting.Source: C Are More Optimization Steps Needed? · p. 17, Figure 7 · p. 18
• Evaluation prompts support functionality, robustness, test-case coverage, and documentation dimensions.Source: Figure 9 · p. 20, Figure 10 · p. 21
• The reviewer prompt evaluates solution code against a specified criterion.Source: C Are More Optimization Steps Needed? · p. 21
• The pipeline includes locator and optimizer prompts for identifying underperforming agents and improving their system prompts.Source: Figure 11 · p. 22, Figure 12 · p. 23, C Are More Optimization Steps Needed? · p. 23