PaperlayerView exact revision on arXiv

Source-linked AI summary

Meta-Moderator: Empowering Multi-Agent Debate with Meta-Cognition

Wentao Hu, Zhuoyue Wan, Jinhao Shen, Chen Jason Zhang, Xiaoyong Wei, Qing Li

arXiv:2608.23029v1cs.CL

TL;DR

Multi-agent debate can improve reasoning, but weak moderation leaves systems vulnerable to redundant deliberation and unreliable evidence aggregation. Meta-Moderator learns an independent policy that monitors and regulates debate before adjudicating a final answer. Across five benchmarks and varied configurations, it improves over common decision layers while adaptively budgeting debate and reducing mis-aggregation.

  • Problem

    Existing multi-agent debate systems rely on fixed budgets, agreement-based stopping, or untrained judges, leaving deliberation control and evidence aggregation unreliable.

  • Method

    Meta-Moderator learns an explicit moderation policy independently from debaters via outcome-driven policy optimization to regulate continuation and final commitment.

  • Results

    Across five reasoning benchmarks and two backbone settings, Meta-Moderator consistently improves accuracy over common debate decision layers and generalizes across tasks, backbones, and debate group sizes.

  • Takeaways & Limitations

    Learnable moderation selectively budgets debate, increases the likelihood that a correct hypothesis emerges, and reduces mis-aggregation after informative hypotheses appear.

  • Takeaways & Limitations

    The approach adds training and inference cost because a dedicated moderator requires extra model calls and outcome-driven reinforcement-learning computation.

Abstract

from arXiv · show

Multi-agent debate can improve large language model reasoning by eliciting diverse hypotheses and critiques, yet its performance is often constrained by weak moderation. Common pipelines rely on fixed budgets, agreement-based stopping, or untrained judges, leading to redundant deliberation and unreliable evidence aggregation. We cast moderation as a meta-cognitive process, monitoring debate utility, controlling deliberation, and adjudicating a final answer, and introduce Meta-Moderator, a learnable framework that dynamically regulates debate and decides when to finalize an answer. Meta-Moderator is trained independently of the debaters via outcome-driven policy optimization, making debate regulation an explicit capability rather than an incidental effect of prompting. Across five benchmarks, Meta-Moderator outperforms widely used decision layers and transfers across tasks and system configurations. Further analyses show that it allocates debate more selectively and reduces mis-aggregation after informative hypotheses appear.

1 Introduction

Multi-agent debate can improve reasoning, but weak moderation causes redundancy, drift, and mis-aggregation. The paper frames moderation as a learnable meta-cognitive loop and introduces Meta-Moderator to regulate deliberation and final decisions.

  • Multi-agent debate improves reasoning by enabling agents to interact, critique one another, and iteratively refine answers across several tasks.
  • Redundancy, drift, and mis-aggregation can make additional debate unproductive or produce incorrect final decisions.These failures arise when systems lack reliable progress assessment, deliberation allocation, and final-decision mechanisms.
  • Moderation is a central bottleneck because it must maintain global context, track resolved uncertainty, and decide when discussion is ready for judgment.
  • Effective moderation combines monitoring progress, controlling whether to continue deliberation, and adjudicating debate content robustly against superficial agreement and aggregation bias.
  • Meta-Moderator learns an explicit moderation policy independently from debaters using outcome-driven reinforcement learning to decide whether to continue or commit.
  • Across multiple reasoning tasks, learned meta-level regulation improves final answer accuracy compared with common multi-agent debate baselines.

2 Related Work

Related work shows that debate gains are inconsistent and that existing stopping and aggregation methods remain brittle. This paper instead studies moderation as a learnable coordination policy over multi-agent deliberation.

  • Multi-agent debate uses proposal, critique, and refinement to improve language-model reasoning, including in multimodal and retrieval-augmented settings.
  • Debate gains may stem from diverse initial answers rather than additional rounds, while conformity and bias amplification can suppress minority hypotheses and propagate errors.
  • Most pipelines use fixed lengths, agreement-based stopping, or untrained judges; measurement-driven alternatives still depend on predefined signals and hand-tuned criteria.
  • Prior metacognition work emphasizes self-monitoring, confidence, uncertainty, and failure prediction, but these signals can be miscalibrated against true knowledge limitations.
  • This paper frames metacognition as coordination: a moderator monitors externalized reasoning traces and regulates whether further debate is worthwhile.
  • Reinforcement learning has been applied to reasoning, tool use, planning, and computation allocation, but moderation of multi-agent evidence remains underexplored.

3 Meta-Moderator

Meta-Moderator separates multi-agent debate from its decision layer, using a learnable policy to monitor debate evidence, regulate continuation, and adjudicate a final answer. It is trained from debate states and deployed as a drop-in controller that can stop adaptively or continue until a maximum horizon.

  • Meta-Moderator: The debate consists of N LLM-based debaters that generate initial responses and iteratively update them using other debaters’ previous-round responses.The transcript H_t records the debate from initialization through round t, while updates occur in parallel.
  • Meta-Moderator: Meta-Moderator models debate moderation as a learnable decision layer that monitors debate utility, controls continuation, and adjudicates the final answer.It replaces static heuristics or prompted judges with a policy over the evolving debate.
  • Meta-Moderator: At each round, the moderator observes a state built from the question and current-round responses, with those responses potentially summarizing prior discussion.This round-level view can encode relevant history without explicitly concatenating the full transcript.
  • Meta-Moderator: The policy treats monitoring as an implicit assessment of whether the current state contains sufficient, non-redundant evidence and uses CONTINUE or STOP to regulate deliberation depth.CONTINUE allocates another debate round, whereas STOP terminates deliberation and triggers final-answer generation.
  • Meta-Moderator: When stopping, the moderator generates a single prediction by aggregating the current debate evidence, enabling adjudication at an intermediate round rather than a fixed horizon.The policy can weigh competing rationales and identify unresolved inconsistencies before committing to an answer.
  • Meta-Moderator: Meta-Moderator is trained on offline debate states using the question, current-round responses, gold answer, and an auxiliary signal indicating whether a correct candidate already appears.The auxiliary signal shapes rewards during training but is unavailable to the moderator at inference time; training uses outcome-driven rewards and GRPO-style optimization.
  • Meta-Moderator: At inference, the system iterates debater generation, state construction, and moderator decisions; it returns the moderator’s answer at STOP or uses the final output at T_max.The moderator is applied as a drop-in regulator for a fixed set of debaters.

4 Experiments

Across five benchmarks, Meta-Moderator generally improves over common MAD decision layers and transfers across tasks, moderator backbones, and debate group sizes. Its full combination of adaptive stopping and learned adjudication performs best, while training reduces mis-aggregation and unnecessary deliberation.

  • Main Results: Meta-Moderator generally achieves the strongest overall performance across five benchmarks, improving over common MAD decision layers.Heuristic majority voting and consensus-based stopping can underperform strong single-agent CoT prompting.
  • Main Results: Meta-Moderator is particularly beneficial on StrategyQA, MATH500, and MMLU, while prompted LLM-as-a-Judge remains competitive on AMC.
  • Generalization: Trained only on GSM8K and MMLU, Meta-Moderator improves accuracy on unseen AMC, MATH500, and StrategyQA benchmarks.The result suggests transfer of debate-utility and decision-readiness signals beyond the training tasks.
  • Generalization: Meta-Moderator remains effective with a smaller moderator backbone and competitive when increasing the number of debaters from N=2 to N=3.Cross-backbone evaluation uses Llama-3.1-8B-Instruct debaters with a Llama-3.2-3B-Instruct moderator; the N=3 evaluation keeps the backbone and debate budget fixed.
  • Ablation: Combining adaptive stopping with learned adjudication performs best across all five benchmarks, whereas either component alone does not consistently recover the full model’s gains.
  • Coverage and Adjudication: 33 to 12, 8 to 1, 25 to 8, 42 to 9, and 77 to 15: trained Meta-Moderator reduces mis-aggregation across GSM8K, AMC, MATH500, StrategyQA, and MMLU.Coverage is not uniformly increased, and lower coverage may partly reflect earlier termination because it is measured at each method’s stopping time.
  • Debate Rounds: Trained Meta-Moderator uses substantially fewer debate rounds while remaining adaptive, reducing redundant debate on easier instances and allocating more rounds on harder benchmarks.The untrained variant often consumes nearly the full maximum budget, especially with the smaller backbone.

5 Conclusion

Meta-Moderator treats moderation as a central bottleneck in multi-agent debate and regulates deliberation through a learned continue-or-stop policy. Across five reasoning benchmarks and two backbone settings, it improves accuracy and supports more reliable, efficient deliberation.

  • Meta-Moderator addresses moderation as a central bottleneck in multi-agent debate.
  • It uses a meta-cognitive loop of monitoring, control, and adjudication to regulate debate and decide whether to continue or commit.
  • Across five reasoning benchmarks and two backbone settings, Meta-Moderator consistently improves accuracy over common debate decision layers and generalizes across tasks and configurations.
  • Analyses show adaptive debate budgeting, a higher likelihood that a correct hypothesis emerges, and reduced mis-aggregation after it does.

Limitations

The approach has computational overhead from training and running a dedicated moderator. Its control interface is limited to continuing or stopping, excluding richer interventions that might improve debate quality.

  • A dedicated moderator adds training and inference costs to multi-agent debate.Monitoring and adjudication require extra model calls, while outcome-driven reinforcement-learning training increases compute relative to prompted decision layers.
  • The moderator’s control interface is restricted to the minimal continue/stop action space.
  • This restricted interface isolates learnable stopping and adjudication but does not capture richer interventions that may further improve debate quality.
  • Future work could reduce overhead through distillation, amortized monitoring signals, or adaptive agent pruning.
  • Future work could expand moderation actions and evaluate objectives beyond final-answer accuracy, including how moderators steer deliberation.

A More Implementation Details

The implementation uses an inference-time moderator with a constrained continue/stop interface and trains it from single-round debate snapshots involving two debaters. The method uses outcome-driven GRPO-style optimization and fixed prompt templates for debate decisions.

  • The GRPO update uses a PPO-style clipped surrogate objective with a KL regularizer, with G=8, ϵ=0.2, β=0, and 3000 gradient-update steps.
  • Inference-time moderation uses an input, a set of debaters, initialization and debate operators, a moderator policy, a state formatter, and a maximum round budget.
  • At each round, the moderator parses either CONTINUE or STOP with an optional final answer, and terminates with a moderator output when the maximum budget is reached.
  • The debate protocol uses prompts that ask debaters to reason step by step, critique other agents’ responses, and place answers in <answer> tags.
  • Training instances are single-round snapshots containing a question, two debater responses, a CONTINUE or STOP label, and, when applicable, a target final answer.
  • The moderator prompt constrains outputs to CONTINUE or STOP, with a tagged final answer required when stopping.

F.1 Training cost

Meta-Moderator requires offline debate-trace generation and reinforcement-learning optimization, while adding one moderator call per inference round. Adaptive early stopping can reduce debater-side generation, making net inference token cost comparable or often lower, but the method is not universally cheaper than prompt-based baselines.

  • 11.25M tokens are used to construct offline training data, comprising 9.78M input and 1.46M output tokens.
  • The moderator is trained once and reused across tasks and backbone configurations, while debaters are not fine-tuned.
  • One additional moderator call per round is typically offset by fewer debate rounds, making net inference token cost comparable and often reduced through adaptive early stopping.
  • The authors do not claim Meta-Moderator is universally cheaper than prompt-based baselines; the claim is improved accuracy–compute trade-off through adaptive deliberation allocation.

G Trained Single-Agent GRPO Baseline

The trained single-agent GRPO baseline tests whether improvements arise from GRPO training alone. Although Naive+GRPO helps on some benchmarks, it remains below Meta-Moderator, indicating added benefits from meta-level debate regulation.

  • The baseline uses Llama-3.1-8B-Instruct, the same GRPO optimizer, and final-answer correctness rewards with dataset-specific matching.Training uses the same splits as Meta-Moderator, with 500 instances each from GSM8K and MMLU.
  • Naive+GRPO improves over naive prompting on some benchmarks but remains below Meta-Moderator.The comparison suggests GRPO training alone does not explain Meta-Moderator’s gains.
  • Meta-level regulation of multi-agent debate provides additional benefits through adaptive stopping and adjudication.

H Additional Results with Different Backbone Settings

Additional backbone experiments show that learned Meta-Moderator performance transfers across different debater and moderator configurations. It consistently improves over heuristic aggregation and remains competitive with or better than an LLM-as-Judge.

  • Three backbone settings vary the debaters and moderator, including Llama-3.2-3B, Qwen-2.5-14B, and Qwen-3-30B debaters.The moderator is either Llama-3.2-3B or Llama-3.1-8B-Instruct, depending on the setting.
  • Learned Meta-Moderator consistently improves over Majority-Voting and Consensus across the additional settings.
  • Meta-Moderator is competitive with or better than an LLM-as-Judge under both reported settings.
  • In the 7B setting, Meta-Moderator yields clear gains on MATH500 and StrategyQA over heuristic aggregation.The authors associate these gains with better debate termination and answer selection.

I Case Study

The case study shows how untrained moderation can continue deliberation after the correct hypothesis appears, allowing later drift to cause mis-aggregation. The trained moderator stops earlier and selects the correct answer.

  • The trained Meta-Moderator recognizes early decision readiness, terminates before drift dominates, and selects B. friendship.
  • Both debaters initially propose B. friendship, so the correct hypothesis is already available in the debate history.The example asks which requirement is greatest for a complete life according to Epicurus; the answer is B. friendship.
  • The untrained moderator continues despite diminishing marginal utility, and a later spurious argument contributes to an incorrect selection.
  • Untrained moderation exhibits over-deliberation and problem drift, where redundant rounds introduce distracting arguments and increase mis-aggregation risk.
  • Figure 4 contrasts a failure by the untrained Meta-Moderator∗ with a successful decision by the trained Meta-Moderator.
Loading 2608.23029v1…