PaperlayerView exact revision on arXiv

Source-linked AI summary

CMAT: A Multi-Agent Collaboration Tuning Framework for Enhancing Small Language Models

Xuechen Liang, Yangfan He, Meiling Tao, Yinghui Xia, Jianhui Wang, Tianyu Shi, Jun Wang, JingSong Yang

arXiv:2404.01663v6cs.CLcs.AIcs.CC

TL;DR

LLM use still depends on human guidance, and large-model deployment faces resource and robustness challenges. The paper presents TinyAgent and CMAT, a multi-agent framework with role coordination and feedback. It reports that tuned small models can match or surpass larger models in some aspects and compete with advanced models on certain tasks.

  • Problem

    LLMs depend heavily on human input to guide dialogue, while computational demands, data bias, and robustness challenges constrain real-world deployment.

    Source: §I · p. 1

  • Method

    The paper introduces TinyAgent and CMAT, which coordinates User, Assistant, and Checker roles and uses environmental feedback for policy refinement.

    Source: §III · p. 2, A. Agent Roles and Actor-Critic Dynamics · p. 2, §I · p. 2

  • Results

    The authors report that carefully trained small models can perform comparably to larger models, with TinyAgent matching or surpassing larger models in some task aspects.

    Source: §V · pp. 5–6, D. Results analysis · p. 5

  • Takeaways & Limitations

    The findings highlight the potential of model design, optimization, and precise tuning strategies for improving the capabilities of smaller models.

    Source: §V · pp. 5–6

Abstract

from arXiv · show

Open large language models (LLMs) have significantly advanced the field of natural language processing, showcasing impressive performance across various tasks.Despite the significant advancements in LLMs, their effective operation still relies heavily on human input to accurately guide the dialogue flow, with agent tuning being a crucial optimization technique that involves human adjustments to the model for better response to such guidance.Addressing this dependency, our work introduces the TinyAgent model, trained on a meticulously curated high-quality dataset. We also present the Collaborative Multi-Agent Tuning (CMAT) framework, an innovative system designed to augment language agent capabilities through adaptive weight updates based on environmental feedback. This framework fosters collaborative learning and real-time adaptation among multiple intelligent agents, enhancing their context-awareness and long-term memory. In this research, we propose a new communication agent framework that integrates multi-agent systems with environmental feedback mechanisms, offering a scalable method to explore cooperative behaviors. Notably, our TinyAgent-7B model exhibits performance on par with GPT-3.5, despite having fewer parameters, signifying a substantial improvement in the efficiency and effectiveness of LLMs.

I. INTRODUCTION

The paper addresses LLMs’ dependence on human guidance and the resource demands of large models. It introduces TinyAgent and CMAT, a collaborative framework with dynamic memory updates and role-based task allocation.

I. INTRODUCTION

  • LLM operation still depends heavily on human input to guide dialogue, prompting the question of whether agents can steer tasks with minimal supervision.The paper also identifies computational demands, data bias, and robustness as deployment challenges, particularly in resource-constrained settings.Source: §I · p. 1
  • TinyAgent models rival advanced LLMs on certain agent tasks, while the figure describes performance comparable to GPT-3.5.The models are fine-tuned from Qwen and CodeLlama.Source: Figure 1 · p. 1, §I · p. 2
  • CMAT uses specialized agents to collaborate on tasks, sharing insights and learning from interaction to support scalable, flexible training.The framework aims to help smaller models bridge performance gaps and adapt to challenges without extensive human intervention.Source: §I · p. 2
  • The framework introduces dynamic, real-time memory updates and a role-playing mechanism for task allocation and agent communication.The authors identify these as main contributions.Source: §I · p. 2

II. RELATED WORK

Prior work describes LLMs as versatile agents that can collaborate across roles, while identifying contextual understanding, sustained memory, and adaptation as challenges. The paper presents short- and long-term memory modes with environmental feedback as a way to support interaction in dynamic environments.

A. LLMs Applications in a Multi-Agent Framework

  • LLMs can act as users, assistants, and checkers, and support communication and coordination in complex multi-agent tasks.Prior work also covers supervised fine-tuning and real-time feedback learning for tasks such as operating systems and databases.Source: A. LLMs Applications in a Multi-Agent… · p. 2
  • Multi-agent settings pose challenges in contextual understanding, sustained memory, adaptation to changing environments, data bias, security, and cooperation protocols.These concerns are identified as hurdles for the domain.Source: A. LLMs Applications in a Multi-Agent… · p. 2
  • The paper implements short-term memory with environmental feedback and long-term support to help LLMs learn from past experience and adapt in dynamic environments.Short-term and long-term memory are presented as modes for improving interaction and adaptation.Source: A. LLMs Applications in a Multi-Agent… · p. 2

B. The tuning method for LLMs

The paper situates CMAT among fine-tuning approaches and describes a three-role framework in which an assistant acts and a checker provides feedback. Its agent workflow uses short-term trajectory context and stores self-reflective outputs as long-term memory.

B. The tuning method for LLMs

  • Supervised fine-tuning and reinforcement learning address different task settings, and the paper notes that high-quality data can let smaller models approach larger models’ performance.Supervised fine-tuning is described as suitable for NLU, while reward-guided reinforcement learning is suited to complex and variable tasks.Source: B. The tuning method for LLMs · p. 2

III. METHODOLOGY

  • CMAT coordinates a User, Assistant, and Checker, combining model fine-tuning with ideas from RLHF, Chain of Thought, and ReAct.The framework is presented as improving decision-making, controllability, and efficiency in complex systems.Source: §III · p. 2

A. Agent Roles and Actor-Critic Dynamics

  • The Assistant generates actions from user input, and the Checker evaluates each action and returns feedback to guide policy refinement.In the depicted workflow, the assistant uses trajectory history as short-term memory and stores self-reflective outputs as long-term memory; the checker verifies SQL before execution.Source: A. Agent Roles and Actor-Critic Dynamics · p. 2, Figure 2 · p. 3

1) Supervised Fine-Tuning:

The Assistant is first fine-tuned on labeled data, then uses Chain of Thought and ReAct to support reasoning and action. The paper reports that generating reasoning steps before the answer improves correctness.

1) Supervised Fine-Tuning:

  • The Assistant is fine-tuned on labeled data using cross-entropy loss before feedback loops are incorporated.This step is intended to help the model produce useful actions.Source: 1) Supervised Fine-Tuning · p. 3

2) Incorporating CoT and ReAct:

  • Chain of Thought generates intermediate reasoning before the final action, while ReAct interleaves reasoning and acting tokens for more context-aware decisions.The paper reports that generating reasoning steps before the final answer improves correctness.Source: 2) Incorporating CoT and ReAct · p. 3

3) Feedback-Driven Policy Optimization:

The Assistant refines its policy using Checker feedback through an Actor-Critic-like scheme without full reinforcement-learning exploration. Feedback supplies the reward, and iterative corrective signals refine decision-making.

3) Feedback-Driven Policy Optimization:

C. Checker-In-The-Loop Mechanism

The Checker is placed directly in the Assistant’s training loop to provide immediate corrective signals. The Assistant also uses short-term memory for recent context and long-term memory for significant insights from self-reflection.

C. Checker-In-The-Loop Mechanism

D. Memory Management and Self-Reflection

IV. EXPERIMENTS

The evaluation tests agents across six task domains and uses a curated dataset built through collection, filtering, enhancement, and distillation. A database example contrasts reflection-enabled behavior with behavior without reflection.

IV. EXPERIMENTS

  • The evaluation covers six domains, including operating systems, database operations, WebShop, knowledge graphs, complex web tasks, and ALFWorld reasoning and visual tasks.Source: §IV · p. 3
  • In the database example, TinyAgent-7B changed its operation after reflecting on a failed attempt, whereas without reflection it repeated the operation and failed.Source: Figure 3 · p. 4

A. Dataset

  • The dataset was self-collected, filtered, enhanced, and distilled, with augmentation and distillation used to address imbalance and insufficient samples.Source: A. Dataset · p. 4

B. Evaluating Code Correction

The experiments evaluate code correction and related agent tasks against selected baselines. The reported results describe strong task-specific performance for TinyAgent models and better performance from high-quality prompts than from low-quality or absent prompts.

B. Evaluating Code Correction

C. Baselines

  • Qwen-1.8B and CodeLlama-7B are selected as baselines to assess the TinyAgent series without CMAT.Source: C. Baselines · p. 4

D. Results analysis

  • TinyAgent-1.8B is reported on par with GPT-3.5 on KG, while TinyAgent-7B surpasses CodeLlama-7B on DB and scores competitively against GPT-4.The authors also report that the CMAT framework can help smaller models compete closely with advanced models.Source: D. Results analysis · p. 5
  • The results analysis reports that CMAT further amplifies the functionality of TinyAgent models refined from Qwen-1.8B and CodeLlama-7B.Source: D. Results analysis · p. 5
  • High-quality prompts outperform low-quality prompts and no prompts across all reported evaluation metrics.Source: D. Results analysis · p. 5

E. Error analysis

The error analysis finds that many agents fail to turn database requests into actual SQL operations, while TinyAgent performs those operations with clearer execution and pre-action analysis. Across six tasks, task-limit overruns are the main cause of incomplete tasks, and the instruction ablation favors combining agent-specific and general instructions.

E. Error analysis

  • Many models acknowledge database instructions without executing SQL, while TinyAgent converts requests into SQL operations and provides clearer execution and pre-action analysis.TinyAgent also adheres to user-specified SQL formats and reflects before acting.Source: E. Error analysis · p. 5
  • Task-limit overruns are the main reason tasks remain incomplete across the six-task execution analysis.The analysis also identifies context-limit exceedances and invalid actions as error types.Source: Table IV · p. 5, E. Error analysis · p. 5
  • The TinyAgent-7B composite performs best in the instruction ablation, while the general-only variant cannot perform KG or ALF.The agent-only variant declines, and the composite is especially effective on WS and DB, supporting the combination of agent-specific and general instructions.Source: E. Error analysis · p. 5

V. CONCLUSIONS

The conclusions report that carefully trained small models can perform comparably to larger models and that CMAT has potential to improve large-model performance. In evaluation, some fine-tuned TinyAgent models reached capabilities similar to API models, whereas most open-source LLMs performed poorly without optimization.

V. CONCLUSIONS

  • Carefully trained small-parameter models can achieve performance comparable to large-parameter models, and CMAT shows potential to improve large-model performance.The authors emphasize model design and optimization strategies in relation to parameter size.Source: §V · p. 5
  • The conclusions include an ablation study of agent and general instructions on TinyAgent-7B.The supplied caption identifies the study's scope but gives no results.Source: Table V · p. 6
  • Some models displayed capabilities similar to API models after meticulous TinyAgent fine-tuning, while most open-source LLMs performed poorly without optimization.The passage does not identify which models or tasks showed similar capabilities.Source: §V · p. 6

Related papers

Loading 2404.01663v6…