PaperlayerView exact revision on arXiv

Source-linked AI summary

Optima: Optimizing Effectiveness and Efficiency for LLM-Based Multi-Agent System

Weize Chen, Jiarui Yuan, Chen Qian, Cheng Yang, Zhiyuan Liu, Maosong Sun

arXiv:2410.08115v2cs.CLcs.AI

TL;DR

LLM-based MAS lack efficient communication and unified parameter-updating methods for optimizing collective performance. OPTIMA trains MAS with iterative selection and balanced rewards, using MCTS-inspired DPO data generation. It consistently improves performance and token efficiency, reaching up to 2.8x performance with 90% lower token usage against baselines.

  • Problem

    LLM-based MAS struggle with inefficient communication and lack parameter-updating methods that optimize the collective system.

    Source: §1 · p. 1

  • Method

    OPTIMA iteratively generates, ranks, selects, and trains interactions using rewards that balance task performance, token efficiency, and communication readability.

    Source: §1 · p. 2

  • Results

    OPTIMA consistently outperforms single-agent and multi-agent baselines, achieving up to 2.8x task performance with 90% lower token usage.

    Source: §1 · p. 2

  • Takeaways & Limitations

    OPTIMA’s efficiency gains may enable more effective inference-compute utilization and improved inference-time scaling laws.

    Source: §1 · p. 2

    Experiments primarily focus on two-agent systems with shared architectures, while scaling to larger teams and heterogeneous agents remains open.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large Language Model (LLM) based multi-agent systems (MAS) show remarkable potential in collaborative problem-solving, yet they still face critical challenges: low communication efficiency, poor scalability, and a lack of effective parameter-updating optimization methods. We present Optima, a novel framework that addresses these issues by significantly enhancing both communication efficiency and task effectiveness in LLM-based MAS through LLM training. Optima employs an iterative generate, rank, select, and train paradigm with a reward function balancing task performance, token efficiency, and communication readability. We explore various RL algorithms, including Supervised Fine-Tuning, Direct Preference Optimization, and their hybrid approaches, providing insights into their effectiveness-efficiency trade-offs. We integrate Monte Carlo Tree Search-inspired techniques for DPO data generation, treating conversation turns as tree nodes to explore diverse interaction paths. Evaluated on common multi-agent tasks, including information-asymmetric question answering and complex reasoning, Optima shows consistent and substantial improvements over single-agent baselines and vanilla MAS based on Llama 3 8B, achieving up to 2.8x performance gain with less than 10\% tokens on tasks requiring heavy information exchange. Moreover, Optima's efficiency gains open new possibilities for leveraging inference-compute more effectively, leading to improved inference-time scaling laws. By addressing fundamental challenges in LLM-based MAS, Optima shows the potential towards scalable, efficient, and effective MAS (https://chenweize1998.github.io/optima-project-page).

1 Introduction

LLM-based multi-agent systems need more efficient inter-agent communication and unified optimization of collective performance. OPTIMA addresses these challenges through iterative training, balanced rewards, and MCTS-inspired data generation, improving task performance while reducing token use.

Motivation

  • LLM-based MAS must reduce inefficient inter-agent communication and optimize collective performance as a unified system.Existing methods often produce verbose exchanges and lack parameter-updating methods designed for the whole MAS.Source: §1 · pp. 1–2

OPTIMA

  • OPTIMA uses an iterative generate, rank, select, and train paradigm with rewards balancing task performance, token efficiency, and communication readability.It combines SFT, DPO, and hybrid variants to induce efficient and effective communication behaviors.Source: §1 · p. 2
  • MCTS-inspired DPO data generation treats conversation turns as tree nodes to explore diverse interaction trajectories efficiently.The approach is designed to produce high-quality paired data for DPO training in multi-agent settings.Source: §1 · p. 2

Implications

  • OPTIMA’s lower inference token usage may enable more effective use of inference compute and improved inference-time scaling laws.The proposed efficiency gains could support more samples within fixed computational constraints.Source: §1 · p. 2

Results

  • Up to 2.8x task performance and 90% lower token usage are reported against single-agent and multi-agent baselines.The evaluation covers information exchange and debate tasks using Llama 3 models.Source: §1 · p. 2

2 OPTIMA: Optimizing Multi-Agent LLMs via Iterative Training

OPTIMA trains LLM-based multi-agent systems through iterative generation, ranking, selection, and training to improve communication efficiency and task effectiveness. It initializes diverse communication patterns and instantiates the framework with iterative SFT, iterative DPO, and a hybrid SFT-DPO procedure.

2.1 Overview

  • OPTIMA iteratively generates, ranks, selects, and trains multi-agent trajectories to improve communication efficiency and task completion.The framework formalizes this process as repeated model updates using task data and selected trajectories.Source: §2.1 · pp. 2–3
  • Its reward balances task performance, token efficiency, and communication readability by combining task, token-penalty, and language-model-loss components.The token term discourages verbose exchanges, while the base-model loss regularizes natural and interpretable communication.Source: §2.1 · p. 3

2.2 Initialization

  • Initialization promotes diverse communication by appending randomly selected format specifications, selecting high-reward trajectories, removing the prompts, and fine-tuning the base model.The resulting starting model can generate diverse communication patterns without explicit format instructions.Source: §2.2 · pp. 3–4

2.3 Instantiation 1: Iterative SFT

  • iSFT repeatedly samples trajectories, selects data above a fixed reward threshold, and applies standard supervised fine-tuning without the initialization format pool.It relies on the communication diversity established by the initialized model while continuing to improve task performance and communication efficiency.Source: §2.3 · p. 4

2.4 Instantiation 2: Iterative DPO

  • iDPO uses Monte Carlo Tree Search-inspired exploration, representing conversation turns as nodes and continuations as edges to generate diverse, high-quality preference pairs.The process expands selected nodes, simulates trajectories, backpropagates rewards, and constructs DPO pairs from shared-ancestry nodes meeting reward thresholds and difference criteria.Source: §2.4 · pp. 4–5

2.5 Instantiation 3: Hybrid Iterative Training

  • iSFT-DPO alternates one iteration of supervised fine-tuning with one iteration of Direct Preference Optimization to combine trajectory learning with comparative preference learning.The hybrid is intended to balance exploration of communication strategies with exploitation of effective patterns.Source: §2.5 · p. 5

3 Experiments

Across information exchange, debate, transfer, inference scaling, ablation, communication, and multi-agent settings, OPTIMA generally improves effectiveness and token efficiency. The experiments also show task-dependent trade-offs, transferable skills, improved scaling behavior, and continued gains with three agents.

3.1 Benchmark Results

  • 38.3% higher F1 (2.8x) on 2WMHQA while using only 10% of MAD’s tokens demonstrates OPTIMA’s strongest information-exchange gain.iSFT-DPO achieves the best performance on multi-hop information-exchange tasks while substantially reducing token usage.Source: §3.1 · p. 5, Table 1 · p. 6
  • OPTIMA improves debate performance and efficiency on ARC-C and MMLU, while matching or slightly trailing SC on MATH and GSM8k with much higher token efficiency.The differing results are attributed to task difficulty and limited training data.Source: §3.1 · pp. 5–6

3.2 How Well Does OPTIMA Generalize?

  • Transferred OPTIMA models outperform baseline multi-agent methods on OOD question-answering tasks, with iSFT more than doubling MAD’s 2WMHQA F1 using 14.6% of the tokens.MATH-trained variants, particularly iDPO, also perform comparably to models directly trained on GSM8k with higher token efficiency.Source: §3.2 · p. 6

3.3 Can OPTIMA Improve Inference Scaling?

  • On GSM8k, GSM8k-trained iDPO matches CoT-SC performance with 88.5% fewer tokens, while OPTIMA’s coverage improves logarithmically as self-consistency sampling increases.Most MATH-trained variants also achieve better GSM8k scaling laws than CoT-SC, except iSFT.Source: §3.3 · p. 7, Figure 3 · p. 7

3.4 How Does OPTIMA Evolve Performance?

  • Removing token regularization substantially increases generated tokens, whereas removing LM loss produces concise but often insufficient and hallucination-prone communication.The complete reward balances task effectiveness, token efficiency, and dialogue quality.Source: §3.4 · p. 7, Table 3 · p. 7

3.5 How Agent Communication Evolves over Optimization Iterations?

  • Across optimization iterations, performance first rises with increased communication, then improves further as token usage gradually decreases and communication becomes more concise and task-oriented.Base models tend to produce verbose, repetitive exchanges, whereas OPTIMA-trained models refine them for efficiency.Source: §3.5 · pp. 7–8

3.6 Can OPTIMA Scale with More Agents?

  • In three-agent information-exchange and debate settings, OPTIMA continues to enhance both task effectiveness and communication efficiency.The two-agent experiments provide a controlled setting, while the three-agent results indicate initial scalability beyond that setting.Source: §3.6 · p. 8

4 Related Work

Prior work demonstrates collaborative and iterative refinement capabilities in LLM systems, but LLM-based MAS still lack comprehensive training methods that jointly address effectiveness and efficiency. OPTIMA targets this gap by optimizing communication and task performance together while examining inference-time scaling.

  • LLM-based MAS have achieved collaborative problem-solving through debate, role-playing, software development, and embodied interaction, but scaling systems introduces efficiency challenges.Source: §4 · p. 8
  • OPTIMA addresses these gaps through joint optimization of communication efficiency and task effectiveness in multi-agent systems.Source: §4 · p. 8
  • Iterative refinement methods train models on self-generated reasoning paths, yet iterative refinement remains underexplored for LLM-based MAS and its effectiveness-efficiency trade-offs.Source: §4 · p. 8
  • Inference-time scaling methods improve reasoning performance but increase computational demands, motivating approaches that improve token efficiency without sacrificing comprehensibility.Source: §4 · p. 8

5 Conclusion

OPTIMA is a training framework for LLM-based MAS that combines iterative training, efficiency-aware rewards, and MCTS-inspired data generation. Experiments report consistent gains in communication efficiency, task performance, inference-time scaling, and transfer within supported settings.

  • OPTIMA significantly enhances communication efficiency and task performance, consistently outperforming single-agent and multi-agent baselines.Source: §5 · p. 8
  • Its key innovations are iterative training, a balanced reward function, and MCTS-inspired data generation for LLM-based MAS.Source: §5 · p. 8
  • OPTIMA improves inference-time scaling and transfers effectively to out-of-distribution tasks within the evaluated scope.Source: §5 · p. 8
  • Scaling OPTIMA to larger models and more complex scenarios remains an open direction for future research.Source: §5 · p. 8

Limitations

The study primarily evaluates two-agent systems with shared architectures and does not establish behavior for larger heterogeneous teams or substantially different application domains. Real-world deployment constraints are also outside the evaluation scope.

  • The experiments primarily focus on two-agent systems with shared model architectures, leaving larger teams and heterogeneous agent configurations unresolved.Preliminary three-agent results are promising, but groups of 5–10 agents may introduce new coordination-efficiency challenges.Source: Limitations · p. 9
  • Cross-task transfer is demonstrated within similar domains such as MATH to GSM8k, while transfer across substantially different areas remains unexplored.The paper specifically identifies transfer from QA to math or coding as an open boundary.Source: Limitations · p. 9
  • The evaluation uses standard benchmarks and does not explicitly address additional constraints that may arise in real-world deployment.Source: Limitations · p. 9

A Inference Scaling Laws on Information Exchange Tasks

The inference-scaling analysis compares OPTIMA variants across debate and information-exchange tasks as sampling increases, while accompanying pseudocode details the framework’s core data and training procedures.

A Inference Scaling Laws on Information Exchange Tasks

  • Coverage scales log-linearly with sampling across OPTIMA variants, while majority-voting accuracy tends to plateau earlier on debate tasks.The plateau suggests that improved answer selection may be needed to exploit generated-response diversity fully.Source: A Inference Scaling Laws on… · pp. 12–13, Figure 4 · p. 13
  • Information-exchange tasks show similar log-linear coverage scaling, but weaker majority-voting accuracy improvements than debate tasks.The paper attributes this difference partly to a majority-voting variant designed for F1 scores, which may not capture partial correctness optimally.Source: A Inference Scaling Laws on… · p. 13, Figure 4 · p. 13
  • Across tasks and variants, consistent coverage scaling indicates room for improvement through increased sampling or more advanced answer-selection strategies.Source: A Inference Scaling Laws on… · pp. 12–13
  • In IE tasks, coverage is defined as the average highest F1 score achieved across generated answers for each instance.Source: A Inference Scaling Laws on… · p. 13

B Additional Pseudo-Codes for OPTIMA Variants

C Case Study on Reward Components Ablation

The loss ablation shows that language-model loss constraints improve communication quality and error correction in multi-agent exchanges. OPTIMA initializes training with diverse, quality-filtered trajectories before fine-tuning the model.

Initialization

Loss ablation

D Case Study on Information Exchange Task

The information-exchange case study shows communication becoming progressively more concise and task-oriented across OPTIMA iterations. This refinement preserves effective structure while removing unnecessary verbosity.

Communication evolution

Iterative supervised fine-tuning

Communication evolution

E Case Study on Debate Task

The debate-task case study shows OPTIMA progressively compressing agent exchanges while preserving the core concept. The example illustrates communication refinement across iterations in a distinct multi-agent setting.

Debate task

Communication evolution

Loss comparison

  • On the film-release example, both trained variants answer Age Of The Dragons, with the language-loss version exposing the two release years.Source: E Case Study on Debate Task · p. 15

Implications

  • The communication improvements are associated with increased task performance and reduced token consumption in the reported quantitative results.Source: E Case Study on Debate Task · p. 15

F Results on Llama 3.2 3B

OPTIMA generalizes beyond the original base model to Llama 3.2 3B. In that smaller model, it improves both efficiency and performance.

Generalization

  • OPTIMA significantly improves both efficiency and performance for Llama 3.2 3B, demonstrating transfer across smaller model sizes.Source: F Results on Llama 3.2 3B · p. 15

G Results on Scenarios with More Agents

Optima improves multi-agent performance and token efficiency in three-agent settings while using iterative data generation, ranking, and training procedures. The appendix details MCTS-based trajectory selection, task-specific training schedules, prompt construction, and hyperparameters.

G Results on Scenarios with More Agents

  • Three-agent experiments sample one information-exchange task and one debate task, with debate agents no longer assigned fixed solver and critic roles.Source: G Results on Scenarios with More Agents · p. 15
  • OPTIMA continues to improve debate performance while significantly reducing token usage, although three-agent information-exchange performance is generally worse than in two-agent settings.The information-exchange degradation is attributed to the more distributed nature of the information, whereas the debate setting retains a performance boost.Source: G Results on Scenarios with More Agents · p. 16

H.1 Data Generation

  • MCTS-based data generation filters candidate conversation nodes by edit-distance diversity, samples among the ten highest-reward candidates, and merges nodes sharing a parent.Source: §H.1 · p. 16

H.2 Ranking

  • The framework ranks trajectories using language-model loss under the base model to constrain conversation readability.Source: §H.2 · p. 17

H.3 Training

  • Training schedules differ by task: debate iterations restart from Llama 3 8B to reduce overfitting, while information-exchange iterations continue from the previous model.For iterative SFT, the loss is computed only on agent conversation rather than the prompt; iDPO trains from the previous iteration using RPO loss.Source: §H.3 · p. 17
  • The hybrid iSFT-DPO procedure uses previous models for information-exchange SFT, initial Llama 3 8B for debate SFT, and the latest SFT model for DPO with standard DPO loss.Source: §H.3 · pp. 17–18

H.4 Hyper Parameters

  • Experiments use six training iterations, task-specific hyperparameters, and prompt tables covering information exchange, mathematical reasoning, ARC-C, and MMLU.The initialization prompt pool contains more than 20 manually filtered formats generated with GPT-4 assistance.Source: §H.4 · p. 18, I Prompts used in Experiments · p. 18, Table 8 · p. 19

iSFT-DPO

  • The MCTS-based DPO algorithm expands nodes, filters trajectory pairs by reward thresholds and shared ancestry, and retains the top 50% of resulting trajectories.Source: iSFT-DPO · p. 24

Related papers

Loading 2410.08115v2…