PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Answer Sharing: TrueMAD for Targeted Peer Engagement in Multi-Agent Debate

Dezheng Bao, Sihang Du, Renhong Huang, Zhizhang Yuan, Yifei Sun, Yang Yang

arXiv:2610.18240v1cs.AI

TL;DR

Broadcast-style MAD exposes peer answers without requiring engagement with particular claims, leaving reliable reasoning vulnerable to suppression and errors without targeted correction. TrueMAD inserts one-to-one, free-form peer advice before self-revision, with the recipient retaining control. Across benchmarks, it improves MAD and DMAD on agent-level and vote accuracy.

  • Problem

    Broadcast-style MAD makes peer answers visible without requiring targeted engagement, so reliable reasoning may be suppressed and specific errors may remain uncorrected.

    Source: §1 · pp. 1–2

  • Method

    TrueMAD is a backbone-agnostic test-time plug-in that adds one-to-one, free-form peer advice before self-revision while leaving revision under the recipient’s control.

    Source: §1 · pp. 2–3

  • Results

    3.62 points in Agent ACC and 3.52 points in Vote ACC over MAD; TrueDMAD exceeds DMAD by 4.72 and 4.06 points, respectively, averaged across benchmarks.

    Source: §4.2 · p. 6

  • Takeaways & Limitations

    The findings suggest effective multi-agent reasoning must move beyond answer sharing toward targeted peer engagement.

    Source: §5 · p. 9

Abstract

from arXiv · show

Multi-agent debate (MAD) aims to improve large language model reasoning by exposing agents to alternative solutions and enabling iterative revision. However, standard broadcast-style MAD makes peer answers visible without requiring agents to engage with a particular peer claim. As a result, reliable reasoning may be suppressed by conflicting or majority responses, limiting achievable performance, while erroneous reasoning may persist and degrade final predictions without actionable feedback. We propose TrueMAD, a backbone-agnostic test-time plug-in that introduces one-to-one, free-form peer advice between answer broadcasting and self-revision. Each source agent responds to a target's current reasoning through support, rebuttal, refinement, or further questions, while the target retains control over how to revise. The same mechanism can augment standard MAD and its improved variants without parameter updates. Across AIME, GPQA-Diamond, and TruthfulQA with Qwen3-30B-A3B, TrueMAD improves agent-level and strict-majority accuracy over MAD by 3.62 and 3.52 percentage points on average, while TrueDMAD improves DMAD by 4.72 and 4.06 points, respectively. Analyses at the utterance, agent-transition, and group levels further show more peer-specific deliberation, more correction-oriented revisions, and more favorable consensus dynamics.

1 INTRODUCTION

Broadcast-only MAD exposes peer answers without ensuring deliberation, leaving reliable reasoning vulnerable to majority pressure and errors without targeted correction. TRUEMAD inserts one-to-one advice before self-revision, and the paper evaluates its effects across tasks and debate dynamics.

1 INTRODUCTION

  • MAD uses multiple agents’ independent answers, exchanged responses, and revisions to support more reliable individual and collective decisions.The introduction situates MAD as a way to draw on complementary perspectives rather than a single reasoning path.Source: §1 · p. 1
  • Standard MAD can suppress correct reasoning under majority pressure while leaving specific errors without targeted feedback.Agents may abandon a correct answer amid conflicting responses, while generic answer sharing may not explain which step or assumption needs correction.Source: §1 · pp. 1–2
  • TRUEMAD adds one-to-one, free-form advice before self-revision, allowing peers to support, rebut, refine, or question a target’s reasoning.The target revises using its own response, broadcast peer answers, and advice addressed to it.Source: §1 · pp. 2–3
  • The paper evaluates TRUEMAD across reasoning benchmarks and examines debate dynamics at utterance, agent-transition, and group levels.The introduction also describes round-wise and difficulty-conditioned analyses, plus call-controlled and cross-model experiments.Source: §1 · p. 3

2 RELATED WORK

Prior work has developed MAD protocols and examined whether their gains reflect substantive deliberation. This paper locates a remaining gap in how agents engage with specific peer reasoning before revision.

Multi-Agent Debate and Protocol Variants

  • MAD variants alter agent diversity, communication, context, or decision-making, yet mostly retain answer exchange without requiring claim-specific feedback before revision.The paper presents TRUEMAD as a general plug-in addressing this interaction gap.Source: §2 · p. 3

Re-examining Interaction in MAD

  • Studies question whether MAD gains arise from deliberation rather than voting, protocol sensitivity, majority influence, or answer instability (Smit et al., 2024).Other work cautions that answer changes may reflect instability and that answer-level consensus can conceal reasoning disagreement (Hao et al., 2026; Wang & Yang, 2026).Source: §2 · p. 3

3 METHODS: TRUEMAD AS A PLUG-IN INTERACTION PROTOCOL

The method frames MAD’s interaction gap as a lack of recipient-directed feedback, then inserts pairwise advice into an existing debate protocol while preserving recipient control over revision.

3.1 MOTIVATION

  • The design goals are targeted feedback conditioned on a recipient’s response, flexible support or correction, and recipient-controlled revision compatible with existing protocols.The plug-in requires no parameter updates, fixed adversarial roles, or external judge at inference.Source: §3.1 · p. 4

3.2 MAD FORMULATION

  • The base MAD protocol initializes N agents, updates their responses across T rounds using peer responses, and aggregates a final group prediction.Its updater and aggregation operator are protocol-specific, covering homogeneous MAD and variants with agent-specific reasoning procedures.Source: §3.2 · p. 4

3.3 TRUEMAD: TARGETED PEER ENGAGEMENT

  • TRUEMAD inserts advice generation between answer broadcasting and self-revision, with sources addressing targets’ current responses through support, rebuttal, refinement, or questions.Advice is generated over ordered source–target pairs without an external router or judge; recipients retain their own and broadcast responses and may accept, reject, reconcile, or partly use advice. The procedure preserves the base protocol and requires no parameter updates.Source: §3.3 · p. 5

4 EXPERIMENTS

The experiments test performance, difficult-case behavior, debate dynamics, inference-budget effects, and cross-model stability. Main comparisons use Qwen3-30B-A3B on AIME, GPQA-Diamond, and TruthfulQA, comparing MAD and DMAD with their TrueMAD-augmented variants under matched settings.

4.1 EXPERIMENTAL SETUP

  • The main setup uses four agents and five response rounds, with matched comparisons holding the backbone, questions, prompts, extraction, and aggregation fixed.The benchmarks are AIME, GPQA-Diamond, and TruthfulQA; the protocols include MAD, promptMAD, DMAD, TRUEMAD, and TRUEDMAD.Source: §4.1 · pp. 5–6

4.2 EXPERIMENTAL RESULTS (RQ1 & RQ2)

  • Averaged across benchmarks, TRUEMAD improves over MAD by 3.62 points in Agent ACC and 3.52 in Vote ACC; TRUEDMAD improves over DMAD by 4.72 and 4.06 points, respectively.Table 1 reports mean ± sample standard deviation over three seeds.Source: Table 1 · p. 6, §4.2 · p. 6
  • TRUEMAD continues improving through R5 on AIME and GPQA-Diamond, with larger gains on initially difficult questions.From R3 to R5, TRUEMAD gains 1.20 points on AIME and 1.18 on GPQA-Diamond, versus 0.46 and 0.33 for MAD. On difficult cases, R5 accuracy gains average 7.13 points for TRUEMAD over MAD and 8.63 for TRUEDMAD over DMAD.Source: §4.2 · pp. 6–7, Figure 2 · p. 7

4.3 DEBATE DYNAMICS ANALYSIS (RQ3)

  • TRUEMAD improves peer-argument engagement, independent reasoning, and novel contribution over matched baselines across benchmarks.Against promptMAD, it scores higher on RP and EI across all three benchmarks, while promptMAD is slightly higher on NC.Source: §4.3 · pp. 7–8, Figure 3 · p. 8
  • CPL rises in all six matched comparisons; on AIME, it increases from 0.85 to 1.96 for TRUEMAD/MAD and from 0.80 to 1.01 for TRUEDMAD/DMAD.The increases indicate more correction-oriented revisions rather than simply more answer changes.Source: Figure 3 · p. 8, §4.3 · p. 8
  • AME and CMP1v3 improve together in every matched comparison, while higher CPL links correction-oriented revisions to more reliable collective outcomes.The authors interpret the joint changes as more readily reconsidering harmful consensus while allowing correct evidence to survive and spread.Source: Figure 3 · p. 8, §4.3 · p. 9

4.4 INFERENCE BUDGET AND PEER ENGAGEMENT (RQ4)

  • RQ4 matches auxiliary-call budgets across ComputeMAD, Self-AdviceMAD, and TRUEMAD, while treating the comparison as supporting rather than definitive causal evidence.The variants use the same number of auxiliary calls and the same revision input structure as TRUEMAD.Source: §4.4 · p. 9

4.5 CROSS-MODEL STABILITY (RQ5)

  • On AIME, TRUEMAD has higher mean accuracy than MAD and TRUEDMAD higher mean accuracy than DMAD on both Qwen3-32B and GPT-4.1-mini.These results provide cross-model evidence for the plug-in on a dense and a closed-source backbone.Source: §4.5 · p. 9

5 CONCLUSION

The conclusion reports improvements for TRUEMAD over MAD and TRUEDMAD over DMAD, alongside more targeted deliberation and correction-oriented dynamics. It also points to auxiliary-call-matched and cross-model comparisons.

5 CONCLUSION

  • The caption identifies Table 2 as an auxiliary-call-matched comparison with Qwen3-30B-A3B and says bold marks the best method; it does not itself state comparison values.Source: Table 2 · p. 9
  • Table 3 presents cross-model results on AIME and marks the best method for each backbone.The supplied passage gives the table scope, not its results.Source: Table 3 · p. 9, §5 · p. 9
  • Across benchmarks, TRUEMAD improves MAD and TRUEDMAD improves DMAD.The conclusion also reports more peer-specific deliberation and correction-oriented revisions.Source: §5 · p. 9

A.1 DATASETS

The appendix describes three evaluation datasets and five debate protocols, including matched comparisons that isolate targeted advice from broadcast-only or generic-prompt approaches.

A.1 DATASETS

  • AIME includes 90 open-ended questions from 2024–2026 high-school mathematics competitions, with integer-valued answers (Dekoninck et al., 2026).It is designed to assess advanced mathematical reasoning, creative problem-solving, and multi-step derivation.Source: §A.1 · p. 12
  • GPQA-Diamond contains 198 expert-validated graduate-level biology, physics, and chemistry questions (Rein et al., 2023).It is described as the highest-quality GPQA subset and assesses deep domain knowledge and rigorous scientific reasoning.Source: §A.1 · p. 12
  • TruthfulQA evaluates truthful and informative answers to misconception-eliciting questions; this evaluation uses 200 questions sampled with seed 0 (Lin et al., 2022).The fixed sample limits evaluation cost.Source: §A.1 · p. 12

A.2 BASELINES

  • MAD broadcasts prior-round peer responses for revision, while promptMAD adds generic critical-evaluation instructions without directed advice (Yin et al., 2023; Du et al., 2024).The promptMAD control tests whether stronger generic instructions, rather than directed interaction, account for gains.Source: §A.2 · p. 12
  • DMAD varies agents’ reasoning strategies while retaining broadcast exchange, using IO, Chain-of-Thought, Step-Back Prompting, and Self-Refine.The strategy citations are Wei et al., 2022; Zheng et al., 2024; Madaan et al., 2023; and Kim et al., 2023.Source: §A.2 · pp. 12–13
  • TRUEMAD adds one-to-one advice grounded in source and target responses before revision; TRUEDMAD applies that stage to strategy-diverse DMAD.The advice can support, challenge, or refine a target’s reasoning; other vanilla MAD components remain unchanged in TRUEMAD.Source: §A.2 · p. 13
  • The primary matched comparisons are TRUEMAD versus MAD and TRUEDMAD versus DMAD; promptMAD controls for stronger generic instructions.Within matched comparisons, the backbone, questions, agent and round counts, answer extraction, and aggregation rule are held fixed.Source: §A.2 · pp. 12–13

B.1 DETAILED DEBATE STAGE ROUND-WISE RESULTS

Tables 4–6 report Qwen3-30B-A3B round-wise agent-level accuracy from R2 to R5, with means and sample standard deviations over three seeds.

B.1 DETAILED DEBATE STAGE ROUND-WISE RESULTS

  • Tables 4–6 report Qwen3-30B-A3B agent-level accuracy from R2 to R5, with each entry giving the mean and sample standard deviation over three seeds.The supplied passages describe the tables but do not provide their cell values.Source: §B.1 · p. 13
  • Table 4 reports round-wise agent-level accuracy (%) on AIME and marks the highest mean in each round.No table values are supplied here.Source: Table 4 · p. 13
  • Table 5 reports round-wise agent-level accuracy (%) on GPQA-Diamond and marks the highest mean in each round.No table values are supplied here.Source: Table 5 · p. 13
  • Table 6 reports round-wise agent-level accuracy (%) on TruthfulQA and marks the highest mean in each round.No table values are supplied here.Source: Table 6 · p. 13

B.2 RESULTS BY INITIAL CORRECTNESS COMPOSITION

The analysis groups questions by first-round correctness and reports R5 agent-level accuracy across initial-composition groups for three benchmarks.

B.2 RESULTS BY INITIAL CORRECTNESS COMPOSITION

  • Questions are grouped by the number of correct agents in R1; reported groups include wrong majority, even split, correct majority, and difficult cases with at least half wrong.The first three groups are mutually exclusive; the difficult-case group combines wrong-majority and even-split cases.Source: §B.2 · p. 14
  • Table 7 reports Qwen3-30B-A3B R5 agent-level accuracy (%) on AIME, stratified by R1 correctness composition.Bold marks the highest accuracy in each column; no values are supplied here.Source: Table 7 · p. 14
  • Table 8 reports Qwen3-30B-A3B R5 agent-level accuracy (%) on GPQA-Diamond, stratified by R1 correctness composition.Bold marks the highest accuracy in each column; no values are supplied here.Source: Table 8 · p. 14
  • Table 9 reports Qwen3-30B-A3B R5 agent-level accuracy (%) on TruthfulQA, stratified by R1 correctness composition.Bold marks the highest accuracy in each column; no values are supplied here.Source: Table 9 · p. 14

B.3 DETAILED PROCESS-LEVEL SCORES

Tables 10 and 11 report the process-level scores visualized in Figure 3, covering peer-deliberation quality, correction preference, and consensus robustness.

B.3 DETAILED PROCESS-LEVEL SCORES

  • Higher values indicate better performance for all six process-level metrics.CPL is unitless; AME is reported in percentage points and CMP1v3 in percent.Source: §B.3 · p. 14, Table 11 · p. 15
  • Table 10 reports Rebuttal Precision, Evidence Independence, and Novel Contribution scores for Qwen3-30B-A3B.The highest score in each column is shown in bold.Source: Table 10 · p. 14, §B.3 · p. 14
  • Table 11 reports Correction Preference Lift, Asymmetric Majority Escape, and Correct-Minority Propagation scores for Qwen3-30B-A3B.The highest score in each column is shown in bold; AME is in percentage points and CMP1v3 in percent.Source: §B.3 · p. 14, Table 11 · p. 15

C PROMPT TEMPLATES

This appendix section specifies prompt templates and protocols for runtime debate updates, targeted TrueMAD advice, dataset-specific output instructions, utterance-level debate-quality judging, and auxiliary calls. It describes what these prompts and calls receive, produce, and how their outputs are used.

C PROMPT TEMPLATES

  • The appendix reproduces the implemented prompt templates verbatim and states that experiments use four agents unless otherwise specified.Each method executes its own initialization; the targeted-advice template is only for generating peer suggestions, after which recipients use their underlying method’s revision procedure.Source: C PROMPT TEMPLATES · p. 15

C.2.2 DMAD INITIAL PROMPTS

  • MAD, promptMAD, and TRUEMAD use a homogeneous initial prompt, whereas DMAD and TRUEDMAD use four role-specific initializers.The DMAD manners are one-step direct generation, one-step chain-of-thought, two-step concept extraction followed by solving, and three-step self-refinement.Source: C PROMPT TEMPLATES · p. 15, §C.2.2 · pp. 16–17

C.1 RUNTIME PLACEHOLDERS AND OUTPUT FORMATS

  • The runtime-placeholder table defines template variables, and dataset-specific instructions request boxed answers for open-ended AIME items or boxed choice letters for multiple-choice GPQA-Diamond and TruthfulQA items.The appendix says brace-enclosed expressions are runtime placeholders replaced with the corresponding question, response, or intermediate output.Source: Table 12 · p. 15, §C.1 · pp. 15–16

C.2.1 MAD-FAMILY INITIAL PROMPT

  • The MAD-family initial prompt consists of the question and dataset-specific output instruction.It is used independently by MAD, promptMAD, and TRUEMAD.Source: §C.2.1 · p. 16

C.3 BROADCAST DEBATE PROMPTS

  • MAD-family debate retains the immediately preceding round; vanilla MAD asks agents to update from others’ answers, while promptMAD explicitly requests evaluation, support, counterarguments, or questions.The two methods use the same debate context, with promptMAD adding the more explicit engagement instruction.Source: §C.3 · p. 17, §C.3.1 · p. 17, §C.3.2 · pp. 17–18

C.4 DMAD DEBATE PROMPTS

  • DMAD debate prompts provide the question, each agent’s previous final answer, and other agents’ previous final answers; intermediate outputs from other agents are excluded.Its four reasoning manners retain one-step IO, one-step CoT, two-step Step-Back, and three-step Self-Refine updates.Source: §C.4 · p. 18, §C.4.1 · p. 18, §C.4.2 · p. 18

C.5 TRUEMAD TARGETED-ADVICE PROMPT

  • TrueMAD generates one comment for each ordered source–target pair, then inserts source-ordered suggestions into the target’s debate context before its unchanged updater revises.The advice prompt uses the question and output instruction plus the source’s and target’s previous answers, and asks for support, counterarguments, or further questions.Source: §C.5 · p. 21

C.6 LLM-AS-A-JUDGE PROMPT FOR LEVEL 1

  • The Level 1 judge scores each debate-stage reply on Rebuttal Precision, Evidence Independence, and Novel Contribution, assigning each dimension an integer from 1 to 10.The rubric separates debate-process quality from answer correctness.Source: §C.6 · p. 21
  • The rubric rewards specific error analysis, evidence-based independence from consensus, and relevant new reasoning, while penalizing generic responses, imitation, repetition, and paraphrase.Source: §C.6 · p. 22
  • The judge evaluates demonstrated reasoning rather than agreement or confidence and must return integer scores in a specified JSON format.Source: §C.6 · p. 22

C.7 AUXILIARY-CALL CONTROL PROMPTS

  • The RQ4 controls use dataset-specific output instructions and identify the current debater through a self-label.The prompt templates use {format_suffix} and {self_label}.Source: §C.7 · p. 22

C.7.1 COMPUTEMAD

  • ComputeMAD receives all four previous-round answers without a comparison target and is invoked 12 times before revisions in R2–R5.Each call generates a non-targeted debate note.Source: §C.7.1 · p. 22
  • Its 12 notes are randomly assigned without replacement, giving each of four agents three notes as advice for its next-round revision.The prompt requests a useful note such as a correction, verification, or missing consideration.Source: §C.7.1 · p. 22

C.7.2 SELF-ADVICEMAD

  • Self-AdviceMAD receives only an agent’s previous-round answer, requests a critique rather than a revised solution, and runs three times per agent before revisions in R2–R5.Its prompt asks for a useful note such as a correction, verification, or missing consideration.Source: §C.7.2 · pp. 22–23
  • Each agent uses its three independently generated Self-AdviceMAD notes as advice for its next-round revision.Source: §C.7.2 · p. 23

Related papers

Loading 2610.18240v1…