ACC-Collab: An Actor-Critic Approach to Multi-Agent LLM Collaboration

• Many reasoning techniques use zero-shot or few-shot capabilities, while multi-agent methods often rely on collaboration emerging in off-the-shelf models.Source: §1 · p. 1
• ACC-Collab jointly trains an actor that answers and a critic that provides feedback to solve problems through iterative conversation.Source: §1 · p. 2
• Guided-Collaboration generates high-quality multi-turn training data for both actor and critic roles.Source: §1 · p. 2
• ACC-Collab significantly outperforms existing state-of-the-art approaches in the authors’ experiments.Source: §1 · p. 2

• Multi-agent deliberation research reports both naturally collaborative abilities [Singhal et al., 2023; Du et al., 2023; Chan et al., 2023] and limits in off-the-shelf models [Wang et al., 2024; Smit et al.].Source: §2 · p. 2
• Multi-agent deliberation methods modify prompts and responses or alter deliberation structure, while generally relying on off-the-shelf models not trained to collaborate.Prompt and response approaches include Liang et al. (2023), Khan et al. (2024), and Rasal (2024); structural approaches include Li et al. (2023a) and Hong et al. (2023).Source: §2 · p. 2
• Unlike debate-data fine-tuning for individual responses, this work trains models to collaborate during both training and inference; adversarial debate trains arguers to win instead.The comparisons are with Subramaniam et al. (2024) and Li et al. (2024b).Source: §2 · p. 2
• Several approaches control disagreement in multi-agent deliberation, often dynamically increasing it to prevent early convergence [Liang et al., 2023; Khan et al., 2024; Chang, 2024a], whereas this study uses opinion changes to generate training data.The study assesses whether opinion changes increase or decrease the likelihood that discussion converges to a correct answer, using that signal to evaluate training examples.Source: §2 · p. 2
• The paper uses model generations as training data, extending self-generated-data work to multiple models debating collaboratively.Related work includes Trung et al. (2024), Huang et al. (2024), and Xiong et al. (2024).Source: §2 · p. 2

• The task-answer pair comes from distribution D; the actor provides answers, the critic provides assistance, and the goal is to infer the correct answer y.Source: §3 · p. 3
• The actor gives an initial response, after which the critic provides feedback and the actor and critic continue alternating responses and feedback over subsequent rounds.Source: §3 · p. 3
• Performance is measured by whether the actor’s final response is correct, with ζ defined as the function that extracts answers from text responses.Source: §3 · p. 3
• The illustrated pipeline generates natural and guided trajectories, selects high-quality pairs by relative quality, and alternates training between actor and critic.Source: Figure 1 · p. 3

4 METHODOLOGY

• ACC-Collab trains an answering actor and a feedback-providing critic to solve tasks through iterative discussion.At inference, the trained agents discuss a task to produce the final response.Source: §4 · p. 3

4.1 AN ACTOR-CRITIC COLLABORATION FRAMEWORK

• A single alternating best-response iteration is sufficient in practice to produce a high-quality collaborative team.The procedure trains the critic and actor in turn, then adapts the critic to the newly trained actor.Source: §4.1 · p. 4

4.2 PARTIAL TRAJECTORY REWARD

• Partial trajectory reward estimates whether deliberation starting from a response at round t will reach the correct answer by round T.Experiments estimate reward by averaging simulated additional rounds using one-step roll-outs; learning-based rewards are left for future work.Source: §4.2 · pp. 4–5

4.3 OFF-POLICY TRAJECTORY GENERATION

• Selecting preference data from many natural rollouts can be difficult when agents perform poorly and computationally expensive even when they perform adequately.This creates a need for a more efficient way to generate positive and negative examples for both agents.Source: §4.3 · p. 6

4.4 GUIDED COLLABORATIVE TRAJECTORIES

• Guided Collaborative Trajectories steer deliberation toward and away from the correct answer, then compare both with natural deliberation to estimate relative goodness.Guided responses are prompted to differ enough from natural responses to affect learning, but not so much that they are difficult to learn; prompt modification is used to strike this balance.Source: §4.4 · p. 6

4.5 LEARNING ON GUIDED TRAJECTORIES

• Direct Preference Optimization trains both agents on positive and negative trajectory pairs, implicitly maximizing the chance that each round leads to a correct final answer.DPO is chosen for efficiency, though the authors state that other preference-optimization schemes could also be used.Source: §4.3 · p. 5, §4.5 · p. 7

5 EXPERIMENTS

• The evaluation uses five benchmark tasks and compares ACC-Collab with inference-based and training-based methods across three base-model families.The tasks are BoolQ, MMLU, BBH, SCIQ, and ARC; the models are Llama-3-8B-Instruct, Mistral-7B-Instruct, and Gemma-2-2B-Instruct.Source: §5 · p. 7

5.1 ACC-COLLAB PERFORMANCE

• Process accuracy, judged by GPT-4o because reasoning-step ground truth is unavailable, improves or is maintained for the actor and critic.Process accuracy evaluates correctness of reasoning and discussion steps rather than only the final answer.Source: §5.1 · p. 7
• ACC-Collab generally exceeds baselines after five deliberation rounds, with one training round often sufficient; additional training can reduce performance.Table 1 reports average accuracy with 95% confidence intervals; the authors note that ACC-Collab+ can underperform ACC-Collab in some cases.Source: §5.1 · p. 7, Table 1 · p. 8

5.2 PERFORMANCE INCREASE OF MULTI-AGENT COLLABORATION

• Across all five datasets, ACC-Collab+ has the highest average improvement for each base model, while several other methods’ gains are less stable.The improvement compares accuracy after five deliberation rounds with a single round.Source: §5.2 · p. 8, Figure 2 · p. 8
• On BoolQ, ACC-Collab variants achieve higher accuracy than other methods at both the initial and final deliberation rounds.At round t = 0, the actor responds independently of the critic.Source: §5.2 · p. 8, Figure 3 · p. 9
• On BoolQ with Llama-3, SFT and DebateGPT can fare little better than SoM-4x, consistent with their focus on raw rather than collaborative accuracy.The authors contrast training for collaboration with training for raw accuracy.Source: §5.2 · pp. 8–9

5.3 INDIVIDUAL PERFORMANCE

• A trained actor paired with an untrained critic usually outperforms SFT and DebateGPT, and pairing it with a trained critic further improves performance.The trained actor attains higher average accuracy than the trained critic when each is paired with an untrained counterpart.Source: §5.3 · p. 9

5.4 WHAT DO THE AGENTS ACTUALLY LEARN?

• Untrained critics are often too agreeable to give substantive corrective feedback, whereas trained critics disagree more readily and provide more detailed feedback.The authors report no notable qualitative change in trained actors’ responses compared with untrained actors’ responses.Source: §5.4 · pp. 9–10

6 CONCLUSION, LIMITATIONS AND IMPACT

• ACC-Collab jointly trains actor and critic agents for iterative discussion, using Guided-Collaboration to generate high-quality preference data.The authors report that ACC-Collab outperforms all baselines across a wide array of domains, and that one training round yields a high-quality team.Source: §6 · p. 10
• After training, critics are more likely to provide detailed disagreements, rather than the agreeable and less verbose responses common without ACC-Collab.Source: §6 · p. 10
• Experiments mainly cover question-answering tasks, train and test within the same task, and use 2B–8B models, leaving other task types, unseen-domain generalization, and larger-scale effectiveness unknown.The method also relies on questions for which correct and incorrect answers can be readily established.Source: §6 · p. 10

REPRODUCIBILITY STATEMENT

• Algorithm 1 gives the data-generation procedure, and Section 4.3 describes the training procedure.Source: REPRODUCIBILITY STATEMENT · p. 11
• Section 5 specifies the datasets, baseline methods, and base models; Section A provides additional experimental details.Source: REPRODUCIBILITY STATEMENT · p. 11
• The prompts are in Section C, and the authors publicly release the code used for the method.Source: REPRODUCIBILITY STATEMENT · p. 11

A EXPERIMENTS

• BBH test and validation sets contain 1,260 and 500 questions, respectively, sampled at roughly 25% and 10% from each category.All reported results use test-set questions.Source: A EXPERIMENTS · p. 15
• Training ran on one Nvidia-H800 GPU; inference used one Nvidia-V100 for Llama-3 and Mistral models and one Nvidia-H800 for Gemma-2.Inference used the VLLM library.Source: A EXPERIMENTS · p. 15
• All models were trained with the trl library using LoRAs of size 256, and ACC-Collab DPO used NLL regularization with weight 1.The cited regularization setup follows Pang et al. (2024a).Source: A EXPERIMENTS · p. 15
• The pipeline generates natural and guided deliberations, selects high-quality pairwise data, trains the actor, and alternates the procedure for actor and critic.Guidance and trajectory-quality calculations differ between the two roles.Source: Figure 5 · p. 15

A.1 CORRECTNESS OF REASONING STEPS

• ACC-Collab typically maintains or increases the accuracy of the actor’s and critic’s reasoning and discussion steps.Table 3 reports these process-accuracy results.Source: §A.1 · p. 16, Table 3 · p. 16
• GPT-4o serves as an oracle for process accuracy because reasoning and discussion steps lack ground-truth answers.The evaluation averages GPT-4o assessments of the steps.Source: §A.1 · p. 16
• The evaluation samples 500 deliberations per team that reached the correct answer after five rounds, then asks GPT-4o to assess their reasoning and discussion steps.The comparison is between untrained and ACC-Collab-trained actor-critic teams.Source: §A.1 · p. 16

B PREFERENCE OPTIMIZATION

• The positive and negative pairs produced by the method can be used directly with indirect-reward preference optimizers such as DPO.The section contrasts these with methods such as PPO that optimize reward directly.Source: B PREFERENCE OPTIMIZATION · p. 16
• For explicit reward maximization, the reward can be learned by simulating deliberation and estimating response values from resampled remaining steps.The learned reward can then be used with the desired preference-optimization scheme.Source: B PREFERENCE OPTIMIZATION · pp. 16–17
• Figures 6–10 plot accuracy over five deliberation rounds on BoolQ, MMLU, BBH, SCIQ, and ARC.The supplied captions do not state the plotted accuracy values.Source: Figure 6 · p. 17, Figure 7 · p. 17, Figure 8 · p. 17

C.1 PROMPTS

• The prompt examples use BoolQ questions based on a passage and ask agents for a brief justification and a yes-or-no answer.The deliberation prompt also presents other agents’ responses for consideration.Source: §C.1 · pp. 17–18
• The single-shot prompt asks for a brief passage-based justification, while its guided version specifies the target answer to support.Both examples omit deliberation among agents.Source: §C.1 · pp. 17–19
• The guided actor prompt supplies a target answer and asks the actor to justify it using the passage and prior responses.The examples include guided deliberation and guided single-shot prompts.Source: §C.1 · pp. 18–19
• The critic prompt asks for brief additional details that could improve the correctness of the actor’s response.The guided critic prompt likewise provides the question, passage, and actor response.Source: §C.1 · p. 19

C.2 CRITIC RESPONSES

• After training, critics were more willing to disagree with the actor and provided more detailed feedback.The section illustrates this qualitative change with responses to wrong actor answers.Source: §C.2 · p. 19
• On the hurricane example, the trained critic points to the 1938 January 3 start as an exception to the actor’s typical-season account.The response uses that historical exception to challenge the actor’s answer.Source: §C.2 · p. 20
• In the severe-initiation example, the trained critic counters the actor’s answer by introducing possible long-term bonding and group cohesion.The example contrasts the actor’s focus on immediate negative effects with the critic’s alternative account.Source: §C.2 · pp. 20–21
• The trained critic also challenges a response about social-complexity sites by distinguishing evidence for agricultural surplus from evidence of complexity alone.The examples contrast Caral’s irrigation evidence with uncertainty about Stonehenge’s relationship to agriculture.Source: §C.2 · p. 21