PaperlayerView exact revision on arXiv

Source-linked AI summary

Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains

Vighnesh Subramaniam, Yilun Du, Joshua B. Tenenbaum, Antonio Torralba, Shuang Li, Igor Mordatch

arXiv:2501.05707v2cs.CLcs.AIcs.LG

TL;DR

LLM self-improvement is limited by training-data constraints and diminishing gains from reduced response diversity. The paper finetunes a society of specialized generation and critic models on independent interaction-generated datasets. This approach preserves diversification, supports improvement over many rounds, and generalizes across reasoning tasks and datasets, while requiring substantially more compute than single-model finetuning.

  • Problem

    LLMs are constrained by their training data, while repeated synthetic-data self-improvement can plateau as response diversity decreases.

  • Method

    The method independently finetunes generation and critic agents initialized from the same base model using datasets generated through multiagent debate and majority voting.

  • Results

    The approach improves performance across a comprehensive suite of reasoning tasks, applies to open-source and proprietary LLMs, and generalizes to novel datasets beyond directly trained baselines.

  • Takeaways & Limitations

    Specialization and diversification across agents allow autonomous improvement over more finetuning rounds than single-agent self-improvement methods.

  • Takeaways & Limitations

    Multiagent finetuning is substantially more expensive than single-model finetuning because multiple model copies must be trained and run.

Abstract

from arXiv · show

Large language models (LLMs) have achieved remarkable performance in recent years but are fundamentally limited by the underlying training data. To improve models beyond the training data, recent works have explored how LLMs can be used to generate synthetic data for autonomous self-improvement. However, successive steps of self-improvement can reach a point of diminishing returns. In this work, we propose a complementary approach towards self-improvement where finetuning is applied to a multiagent society of language models. A group of language models, all starting from the same base model, are independently specialized by updating each one using data generated through multiagent interactions among the models. By training each model on independent sets of data, we illustrate how this approach enables specialization across models and diversification over the set of models. As a result, our overall system is able to preserve diverse reasoning chains and autonomously improve over many more rounds of fine-tuning than single-agent self-improvement methods. We quantitatively illustrate the efficacy of the approach across a wide suite of reasoning tasks.

1 INTRODUCTION

LLM self-improvement can plateau as repeated synthetic-data finetuning reduces diversity. Multiagent finetuning addresses this by specializing diverse generation and critic models, enabling improvement across more rounds and tasks.

  • LLMs are constrained by their training data, motivating self-improvement through additional synthetic training data.
  • Frontier-model supervision is limited by model quality, inference cost, and legal restrictions on commercial-grade models.
  • Repeated self-improvement often plateaus because response diversity decreases after successive finetuning rounds.
  • Multiagent finetuning independently specializes models from a shared base using data generated through multiagent interactions.
  • Distinct generation and critic roles create a feedback loop for producing and refining responses through multiagent debate.
  • The system improves over more finetuning rounds, demonstrates gains across reasoning tasks, and generalizes to novel datasets beyond baselines trained directly on them.

2 MULTIAGENT FINETUNING OF LANGUAGE MODELS

The method uses debate and majority voting to construct role-specific finetuning datasets, then repeatedly updates generation and critic agents. Independent data and correction-focused critic training preserve diversity while enabling iterative refinement.

  • Multiagent debate: Multiagent debate generates responses over multiple rounds, and majority voting determines the final predicted output for each task.
  • Motivation: A single-model finetuning loop can plateau because accurate debate results often share similar styles and methodologies.
  • Finetuning multiple models: Multiagent finetuning creates separate datasets for separate models so the resulting agents specialize and diversify their responses.
  • Iterative procedure: The procedure initializes copied generation and critic agents, iterates debate and finetuning, and then uses the updated agents for continued debate.
  • Generation models: Generation models are finetuned on each agent’s outputs that match the final debate result, producing distinct input-output datasets.
  • Critic models: Critic models evaluate generation outputs and synthesize or select responses, adding feedback beyond generation-model finetuning alone.
  • Critic models: Critic datasets include trajectories that either correct an initially wrong answer or maintain an initially correct answer through the final debate round.
  • Iterative procedure: Iterative multiagent finetuning enables progressively more accurate responses through continuous learning and adaptation.

3 EXPERIMENTS

The experiments evaluate multiagent finetuning and baselines on arithmetic, GSM, and MATH reasoning tasks, using fixed finetuning and evaluation sets. Results show stronger accuracy, sustained gains across iterations, and preserved diversity compared with single-agent finetuning.

  • Evaluation setup: The evaluation covers Arithmetic, Grade School Math, and MATH reasoning tasks, with 500 examples selected for finetuning and 500 held out for evaluation.Accuracy is computed from parsed answer correctness, with standard error reported for each value.
  • Quantitative results: Our method outperforms the Base, Majority, Debate, STaR, and Majority FT baselines across the reported datasets.Table 1 reports accuracy (%) ± standard error; the comparison uses three open-source models and GPT-3.5.
  • Quantitative results: With only 500 finetuning examples, the method shows particularly significant improvement over baselines on challenging GSM and MATH datasets.Additional evaluations on a larger set of problems and datasets are reported in the appendix.
  • Multiple iterations of finetuning: 58.8% to 66.0% accuracy for Phi-3 and 22.5% to 28.2% for Mistral are reported after five finetuning iterations on MATH.The five-round method is reported as 12.6% and 9.31% more accurate than the best Table 1 baseline for Phi-3 and Mistral, respectively.
  • Multiple iterations of finetuning: Single-agent finetuning saturates after one iteration and declines afterward, while multiagent finetuning consistently improves performance over five iterations.The paper attributes the single-agent decline to fixation on a small range of generated responses and potential overfitting.
  • Multiple iterations of finetuning: Multiagent finetuning preserves diversity for one measure and improves it for another across iterations, whereas single-agent diversity drops significantly.Figure 3 measures response diversity on MATH using two diversity measures.

4 ANALYSIS

The analysis studies which components drive performance, how multiagent finetuning affects response diversity, and whether gains generalize across datasets. Results indicate that the integrated framework improves performance while preserving diversity across finetuning rounds.

  • Ablation studies: Single-agent finetuning can cause model collapse by producing similar responses, reducing diversity and hurting performance.
  • Ablation studies: Summarization, critic-generation roles, multiagent finetuning, and multiagent debate each contribute to performance improvement.
  • Agent response diversity: Multiagent finetuning produces more diverse responses than single-agent finetuning across rounds, with increasing NLL indicating greater divergence.
  • Agent response diversity: Improved response diversity correlates positively with improved performance across finetuning rounds for both Phi-3 and Mistral models.
  • Zero-shot generalization: When trained on MATH, the proposed method generalizes zero-shot to GSM and outperforms baselines trained on GSM.

5 RELATED WORK

Related work categorizes finetuning into human-in-the-loop, distillation, and self-improvement, while positioning this paper at the intersection of self-improvement and multiagent interaction. Prior multiagent methods, especially debate, motivate the paper’s interaction-based approach.

  • Finetuning methods are grouped into human-in-the-loop, distillation, and self-improvement categories.
  • Human-in-the-loop and distillation: Human-in-the-loop methods include RLHF and DPO, which have been used to improve instruction-following responses.
  • Human-in-the-loop and distillation: Distillation uses a larger, highly capable language model to generate data for finetuning a smaller model.
  • Self-improvement: Self-improvement methods iteratively finetune language models using generated data, including iterated learning and rationale bootstrapping.
  • Multiagent interaction: The paper builds on multiagent interaction systems, primarily multiagent debate, which has improved factuality and reasoning at inference time.

6 CONCLUSION AND LIMITATIONS

The paper concludes that multiagent finetuning improves language-model performance and diversity through agents with distinct roles, while remaining applicable across model types and reasoning tasks. Its main practical limitation is substantially higher training and inference cost than single-model finetuning.

  • Limitations: Multiagent finetuning is substantially more expensive than single-model finetuning because multiple model copies must be trained and run.Open-source experiments used four H100 or four A100 GPUs, with 120GB–240GB of GPU memory and 12–24 hours of inference.
  • Conclusion: Multiagent finetuning improves performance and diversity by using a society of agents with distinct roles and stronger feedback.
  • Conclusion: The framework supports autonomous iterative self-improvement with substantial gains across a comprehensive suite of reasoning tasks.
  • Conclusion: The approach applies to both open-source and proprietary language models and can be integrated with RLHF or DPO in future work.

B METHODOLOGY DETAILS

The methodology uses summarization within multiagent debate to condense other agents’ responses before critic refinement, then applies a repeated debate procedure and majority voting to produce final predictions. The inference algorithm initializes generation agents, iterates debate rounds, and measures accuracy against ground truth.

  • Debate summarization: A summarization agent receives the previous-round responses from the other N −1 agents and produces a summary for a critic agent.
  • Inference procedure: The inference procedure requires finetuned generation agents, finetuned critic agents, a test set with ground-truth responses, N agents, and M debate rounds.
  • Inference procedure: At the first round, generation agents produce responses directly from each input.
  • Inference procedure: In later rounds, critic agents generate responses from summarized outputs produced from other agents’ responses.
  • Inference procedure: The final prediction is obtained by majority voting over the last-round responses, and accuracy counts predictions matching the ground truth.

B.3 EXPERIMENTAL DETAILS

The experiments use full-model finetuning across several open-source and proprietary LLMs, with model-specific hardware, epoch, batch-size, and learning-rate settings.

  • Compute: Open-source models are finetuned using eight 40GB A100 GPUs and four 80GB H100 GPUs.Evaluating individual multiagent inference times took approximately 30 to 36 hours.
  • Phi-3: Phi-3-Mini-128K-Instruct is finetuned end-to-end for two epochs for generation agents and one epoch for critic agents.Generation agents use learning rate 5e−6, while critic agents use 5e−7; batch size is 1.
  • Mistral: Mistral-7B-Instruct-v0.2 is finetuned end-to-end for one epoch with batch size 1 and learning rate 5e−7 for both agent types.Training uses weight decay 1e−2 and four 40GB A100 GPUs or two 80GB H100 GPUs.
  • LLaMA-3: Meta-Llama-3-8B-Instruct is finetuned end-to-end for two epochs with batch size 1.Generation agents use learning rate 5e−7 and critics 2e−7; training uses three 80GB H100 GPUs.
  • GPT-3.5: GPT-3.5-Turbo-0613 is finetuned for two epochs on 500 randomly selected questions and evaluated on a separate test set of 500 questions.The finetuning API uses batch size 1 and learning-rate multiplier 1, without answer annotations in training questions.

C DIVERSITY METRICS

The paper evaluates response diversity using consensus, KL-divergence, and embedding-based measures, finding that multiagent finetuning preserves diversity across rounds and exceeds single-agent baselines.

  • Consensus: Multiagent response diversity remains consistent across finetuning iterations, while single-agent diversity drops significantly on MATH.Consensus measures the proportion of agents matching the majority-voted final answer; diversity is defined as 1 − Consensus.
  • KL-Divergence: Multiagent finetuning maintains higher KL-divergence than the single-agent finetuning baseline across iterations.KL-divergence is averaged across pairs of agents using response probability distributions estimated with Gemma-2 (2B).
  • Base-model divergence: Both generation and critic agents show higher KL-divergence from the base LLM across finetuning iterations, with critics generally diverging more.The comparison averages KL-divergence over 500 MATH examples and uses Gemma-2 (2B) likelihoods.
  • Embedding Dissimilarity: Embedding dissimilarity also indicates that diversity is preserved across finetuning rounds.Responses are represented using [CLS] embeddings from the T5-3B encoder, with diversity computed as the complement of cosine similarity.
  • Temperature comparison: Increasing sampling temperature does not produce diversity that is as useful for accuracy as the proposed method.The method outperforms a single-agent finetuning baseline evaluated at temperature 2.

D COOPERATIVE FINETUNING

Cooperative multiagent finetuning extends the approach beyond competitive debate, while role- and data-specific specialization supports diverse agents and continued improvement.

  • Cooperative setting: Cooperative finetuning prompts agents to merge their solutions with other agents’ responses instead of critiquing them.The cooperative setting uses multiagent interaction in a mixture-of-experts style and improves performance over Cooperative (Base).
  • Cooperative setting: The cooperative method achieves similar performance improvements and can be applied to other multiagent prompting settings.These results are reported in Table 3.
  • Unique IDs: Multiagent finetuning significantly outperforms the unique-ID approach on the same 500 MATH examples.Unique IDs assign generation or critic identities in prompts, while the full method uses debate; the unique-ID approach performs fairly similarly to debate in the reported comparison.
  • Scaling agents: Using five agents and two debate rounds improves results over using three agents and two debate rounds.The comparison suggests that the method can scale to larger numbers of finetuned agents.
  • Mathematical model: The mathematical model trains each model on questions it answers correctly, increasing topic-specific skill and producing specialization across models.Random initial topic preferences can lead models to specialize in different topics; skills are normalized to reflect finite model capacity.

H.1 LARGER MATH EVALUATION

Additional evaluations test multiagent finetuning on harder MATH problems and on MMLU, finding consistent improvement on both mathematical reasoning and factuality-related tasks.

  • All-levels MATH: 500 training and 500 test examples are sampled across all five MATH difficulty levels for LLaMA-3.This evaluation broadens the task beyond the selected lower-difficulty levels.
  • All-levels MATH: Multiagent finetuning shows consistent improvement across multiple rounds on the all-levels MATH evaluation.Results are reported across baselines in Table 6 and across finetuning rounds in Figure 11.
  • MMLU: On MMLU, the method is finetuned on 500 examples sampled across all 57 subjects and tested on 500 different examples.The evaluation targets a task related to general factuality and reasoning.
  • MMLU: The method improves performance on MMLU relative to other baselines.The reported evaluation uses the separate 500-example test set described above.

H.3 ZERO-SHOT GENERALIZATION EVALUATION

The method is evaluated for zero-shot generalization from MATH or arithmetic finetuning to GSM problems. In both settings, it performs significantly better than all other baselines.

  • Finetuning on 500 MATH problems and evaluating on 1000 GSM problems, the method performs significantly better than all other baselines.
  • Finetuning on 500 arithmetic problems and evaluating on 1000 GSM problems, the method also performs significantly better than all other baselines.
  • Across 1000 GSM problems, the method performs better than all baselines after training on MATH.
  • Arithmetic finetuning improves GSM performance more than finetuning with MATH in the reported comparison.
Loading 2501.05707v2…