Source-linked AI summary
A Picture Is Worth a Graph: A Blueprint Debate Paradigm for Multimodal Reasoning
Changmeng Zheng, Dayong Liang, Wengyu Zhang, Xiao-Yong Wei, Tat-Seng Chua, Qing Li
TL;DR
Multimodal debate can trivialize opinions and divert attention when bottom-up discussion summarizes diverse, image-derived concepts. BDoG confines debate to a blueprint graph and stores evidence in branches; experiments report state-of-the-art results on ScienceQA and MMBench.
Problem
Bottom-up multi-agent debate can trivialize opinions through summarization and divert focus toward irrelevant or correlated concepts in multimodal reasoning.
Source: §1 · pp. 1–2Method
BDoG uses a blueprint graph to confine debate and stores conclusions and newly introduced concepts in relevant branches.
Source: §1 · p. 2Results
BDoG establishes state-of-the-art results on ScienceQA and MMBench, surpassing previous methods.
Source: §6 · p. 9Takeaways & Limitations
The study presents BDoG as a pilot application of multi-agent debate to multimodal reasoning, with graph branches supporting tracking of discussion progress.
Source: §1 · p. 2, §6 · p. 9Blueprint initialization is constrained by size and relevance requirements, including a maximum of 20 graph objects.
Source: §4.1 · p. 4
Abstract
from arXiv · showhide
This paper presents a pilot study aimed at introducing multi-agent debate into multimodal reasoning. The study addresses two key challenges: the trivialization of opinions resulting from excessive summarization and the diversion of focus caused by distractor concepts introduced from images. These challenges stem from the inductive (bottom-up) nature of existing debating schemes. To address the issue, we propose a deductive (top-down) debating approach called Blueprint Debate on Graphs (BDoG). In BDoG, debates are confined to a blueprint graph to prevent opinion trivialization through world-level summarization. Moreover, by storing evidence in branches within the graph, BDoG mitigates distractions caused by frequent but irrelevant concepts. Extensive experiments validate that BDoG is able to achieve state-of-the-art results in ScienceQA and MMBench with significant improvements over previous methods. The source code can be accessed at https://github.com/thecharm/BDoG.
1 INTRODUCTION
Multimodal reasoning must integrate diverse semantics while maintaining logical consistency, but multi-agent debate can trivialize opinions and divert focus. BDoG confines debate to a blueprint graph, preserving specific concepts and organizing new evidence in relevant branches.
1 INTRODUCTION
- Multi-agent debate offers a way to integrate perspectives from language models, but multimodal reasoning still faces the challenge of maintaining logical consistency across diverse semantics.The paper frames multimodal reasoning as requiring both unified semantic representation and integration across modalities.Source: §1 · p. 1
- Multi-agent debate can trivialize opinions through round-end summarization and divert focus toward correlated concepts introduced during chain-of-thought reasoning.The paper illustrates how broad summaries can shift attention from specific concepts to general ones.Source: §1 · p. 1
- Bottom-up debate gathers word-level opinions across disparate concepts, while image-rich multimodal inputs increase semantic divergence and the likelihood of distraction.The paper argues that chain-of-thought amplification of biased concepts can further redirect focus.Source: §1 · pp. 1–2
- BDoG builds an initial graph from modality concepts and relations, then uses it as a blueprint to confine debate and mark conclusions top-down.Specific concepts remain on the graph rather than being merged into general ones.Source: §1 · p. 2
- New concepts are added to relevant branches, shifting semantic competition from words to branches and supporting discussion-progress tracking.The paper presents this structure as reducing focus diversion and increasing explainability.Source: §1 · p. 2
2 RELATED WORKS
Related work covers multimodal reasoning methods, multi-agent collaboration and debate, and graph-augmented language models. BDoG is positioned as an effort to extend debate to multimodal reasoning and to update graphs dynamically through debate.
2.1 Multimodal Reasoning
2.2 Multi-agent Debate
- Self-correction uses feedback to refine predictions [25] [22], while multi-agent systems support collaboration and debate across contrasting perspectives [3] [7] [17].The paper presents its work as an initial effort to extend multi-agent debate to multimodal reasoning and address opinion trivialization and focus diversion.Source: §2.2 · pp. 2–3
2.3 Graph-augmented LLMs
- Graph-augmented methods use knowledge graphs for reasoning, but BDoG differs from static-graph approaches by iteratively updating entities, attributes, and relations through blueprint debate.Prior work includes query generation [16], structured QA [28], reasoning over knowledge-graph triples [26], scene graphs [23], and external knowledge graphs [24].Source: §2.3 · p. 3
3 PRELIMINARY
The preliminary section defines multimodal reasoning as answering a question using visual context and textual clues, then outlines prompting, rationale-based, self-correction, and debate approaches. These descriptions establish the comparison methods and inputs used in the paper’s framing.
3 PRELIMINARY
- Vanilla prompting predicts an answer from a question, visual context, textual clue, and illustrative examples.The task is to select the correct answer from a candidate set.Source: §3 · p. 3
- Multimodal CoT generates rationales that are combined with the language input, while DDCoT aggregates answers to a sequence of sub-questions.DDCoT obtains the sub-question sequence in one interaction.Source: §3 · p. 3
- Self-correction iteratively refines model outputs using feedback, whereas MAD uses multi-round exchanges in which a proponent revises its rationale and response in light of an opponent’s prior rationale.The section characterizes MAD as discursive exchange between conversational models.Source: §3 · p. 3
4 BLUEPRINT DEBATE ON GRAPH
BDoG uses a top-down, graph-structured debate that iteratively refines an initial blueprint, organizing agents’ opinions and proposals to help minimize opinion trivialization and focus diversion.
4 BLUEPRINT DEBATE ON GRAPH
- BDoG structures multimodal debate around a blueprint graph that agents iteratively refine, rather than inducing answers from word-level thoughts.The graph structures agents’ opinions and proposals and the approach integrates diverse perspectives through competition and cooperation.Source: §4 · p. 3
4.1 Blueprint G0 Initialization
- The initial graph is built from textual and visual sources, with size and relevance constraints intended to keep its clues useful and focused on the question.Textual sources include the question and context; the visual source is the image.Source: §4.1 · pp. 3–4
4.2 Agents and Roles
- Proponents refine and defend the blueprint, opponents challenge it, and a moderator synthesizes their evidence graphs; roles are assigned in balanced numbers, with one moderator.The proponent and opponent functions follow size, relevance, and compactness constraints.Source: §4.2 · pp. 4–5, §4.3 · p. 5
4.3 Debate Progress and Graph Condensation
- Debate proceeds in rounds as the moderator combines affirmative and negative graphs into a conclusion graph and tentative answer, while graph modifications can indicate when evidence has condensed.If debate continues, the negative graph becomes the next round’s blueprint; if it concludes, the moderator’s answer is adopted as final.Source: §4.3 · p. 5
5 EXPERIMENTS
The experiments evaluate BDoG across multimodal backbones and two QA benchmarks, report performance against prior methods, and examine component contributions, debate progress, and efficiency.
5.1 Backbone Models
- The evaluation uses GeminiProVision, InstructBLIP, LLaVA-v1.5, and GPT-4 backbones on ScienceQA-IMG and MMBench under zero-shot settings.ScienceQA-IMG uses TEST and DEV partitions; MMBench uses the official MMBench-Dev split and accuracy metric.Source: §5.1 · p. 5, §5.2 · p. 5
- BDoG is also evaluated with InstructBLIP and LLaVA-v1.5, described as more resource-constrained backbones than alternative architectures.The implementation additionally includes GeminiProVision and GPT-4.Source: §5.1 · p. 5
5.3 Performance Comparison to SOTA Methods
- With GeminiProVision, BDoG reaches 81.1% accuracy on ScienceQA-IMG test and 81.3% on MMBench-Dev, with gains of 4.3%–5.7% and 6.1%–11.8% across backbones, respectively.The authors report state-of-the-art performance on both benchmarks; Table 1 presents overall zero-shot results.Source: §5.3 · p. 6, Table 1 · p. 6
5.4 Ablation Study
- Debate-only and graph-only variants contribute differently: debate improves natural-science questions, while graph regulation helps logical and attribute reasoning; combining them improves performance across nearly all categories.The debate-only variant can suffer from trivialization and focus diversion, while graph-only reasoning lacks iterative refinement; Table 2 reports the category-level ablation.Source: §5.4 · pp. 6–7, Table 2 · p. 7
5.6 Efficiency Analysis
- On GeminiProVision, BDoG uses approximately 50% less inference time and achieves over 5 percentage points higher accuracy than vanilla debate across three test sets.Figure 5 compares inference time with zero-shot accuracy.Source: Figure 5 · p. 8, §5.6 · p. 8
5.5 Monitoring The Debating Progress
- Performance generally converges by the second or third debate round, while graph-change statistics track updates, additions, and pruning as debates progress.Moving from one round to two yields notable gains for smaller models, with more modest gains for larger models.Source: §5.5 · p. 8, Figure 4 · p. 8, Table 3 · p. 8
6 CONCLUSION
The paper introduces a pilot study of multi-agent debate for multimodal reasoning and proposes BDoG to address opinion trivialization and distraction from irrelevant concepts. Experiments in ScienceQA and MMBench report new state-of-the-art results.
6 CONCLUSION
- BDoG introduces multi-agent debate into multimodal reasoning as a pilot study.Source: §6 · p. 9
- BDoG confines debate to a blueprint graph and stores evidence in branches to address opinion trivialization and irrelevant-concept distraction.Source: §6 · p. 9
- Experiments in ScienceQA and MMBench report that BDoG surpasses previous methods and establishes state-of-the-art results.Source: §6 · p. 9
A EXPERIMENTAL DETAILS
The experimental appendix outlines BDoG’s iterative graph-debate procedure, in which agents update and condense a shared blueprint. It also describes relevance-based pruning and graph condensation as ways to refine the information used in debate.
A EXPERIMENTAL DETAILS
- Agents begin with an extracted blueprint and iteratively generate affirmative and negative graphs until the graphs match or the maximum debate round is reached.The algorithm takes a question, image, context, multimodal LLM agents, and a maximum-round setting as input.Source: §A.1 · p. 10
- BDoG uses a hierarchical tree in which leaf nodes propose updates to a shared knowledge graph and interior nodes aggregate information flows.Source: §A.1 · p. 10
- Graph condensation revises a large graph into a smaller informative one, treating error correction and irrelevant-information filtering as multimodal reasoning.Entity and relation updates identify content shared by the agents’ graphs.Source: §A.1 · p. 10
- After identifying agreed entities and relations, the LLM judges remaining graph content for relevance and removes items deemed irrelevant.The pruning decision is based on problem context and debate relevance rather than a rigid mathematical operation.Source: §A.1 · p. 10
A.2 Statistics of Datasets
The appendix reports dataset statistics, implementation settings, and inference-time observations, then details BDoG’s agent roles, blueprint extraction, debate updates, and answer generation.
A.2 Statistics of Datasets
- Table 4 summarizes the ScienceQA test and development sets and the MMBench development set by instance counts, subjects, categories, and average question length.Source: §A.2 · p. 10, Table 4 · p. 10
A.3 Model Deployment
- InstructBLIP deployment uses two NVIDIA A100 GPUs, Vicuna-13B, and EVA-CLIP, with output limits for graph generation and candidate answers; MMBench inference takes longer than ScienceQA despite shorter questions.The text suggests more complex MMBench questions and more intricate output graphs as a plausible explanation; GeminiProVision and GPT-4 use official APIs without pre- or post-processing.Source: §A.3 · pp. 10–11, Figure 6 · p. 11, Table 5 · p. 11
A.4.1 Role Specification.
- BDoG assigns agents distinct roles, including diligent explanation, creative problem-solving, and attentive analysis, to encourage exchange of different perspectives.The role prompts combine core activities with individual traits and goals.Source: §A.4.1 · pp. 11–12
A.4.2 Blueprint Extraction.
- Blueprint extraction builds a unified graph of question-relevant objects, attributes, and relations, avoiding the error propagation risk of separately generating captions and semantic associations.The graph represents coarse-grained information without requiring manually annotated scene graphs or image captions.Source: §A.4.2 · p. 12
A.4.3 Proponent and Opponent.
- Proponent and opponent agents update, add, or prune graph nodes; the final answer is extracted directly from their final graphs rather than from iterative text summaries.The two agents can receive each other’s updated graph as their debate input.Source: §A.4.4 · p. 12, §A.4.3 · p. 12
A.4.4 Moderator.
- The moderator uses the image and both debate graphs to answer, keeping visual elements integrated throughout reasoning so textual output remains grounded in visual context.Source: §A.4.4 · p. 12
B QUALITATIVE EXAMPLES
The qualitative examples compare BDoG with vanilla multi-agent debate on ScienceQA-IMG and MMBench. They illustrate fact-level comparison in a map question and image-grounded attribute reasoning in a fossil question.
B QUALITATIVE EXAMPLES
- BDoG is described as more structured, visual, and interactive than vanilla MAD in examples from ScienceQA-IMG and MMBench.Figure 7 uses green for correct answers or rationales and red for incorrect or irrelevant predictions.Source: B QUALITATIVE EXAMPLES · p. 12, Figure 7 · p. 13
- In the map example, BDoG derives the southernmost-state answer by comparing candidate latitudes, making changes easier to monitor.The affirmative agent predicts the relative position correctly, while the negative agent opposes it.Source: B QUALITATIVE EXAMPLES · pp. 12–13
- In the fossil example, BDoG restricts reasoning to observed fossil attributes and reaches “Long legs,” whereas MAD’s added speculation leads to an erroneous live-bear inference.Source: B QUALITATIVE EXAMPLES · p. 13
C.1 Monitoring the Debate Progress
BDoG debate graphs make debate progress inspectable through structural changes, and the reported changes align with diminishing disagreements and errors across rounds.
C.1 Monitoring the Debate Progress
- BDoG outputs JSON debate graphs whose evolving attributes, entities, and relations can be compared with the original graph to quantify debate changes.Source: §C.1 · p. 13
- As debate progresses, disagreements and errors diminish, consistent with the authors’ hypothesis.Source: §C.1 · p. 13
- Figures 8 and 9 report intra- and inter-round counts of updated attributes, pruned entities or relations, and added entities or relations for ScienceQA-IMG dev and MMBench-dev.Source: Figure 8 · p. 13, Figure 9 · p. 14
C.2 Effect of Blueprint Quality
Final correctness is associated with initial blueprint quality, while BDoG also produces correct predictions for many cases with low-quality blueprints.
C.2 Effect of Blueprint Quality
- 56% of 200 sampled MMBench predictions had low-quality initial blueprints, classified by three annotators because no standardized graph-quality metric existed.Source: §C.2 · p. 14
- 93.2% final correctness for high-quality initial blueprints, while 67.8% of low-quality-blueprint cases still end in correct predictions.Source: Figure 10 · p. 14, §C.2 · p. 14
C.3 Effect of Introduced Options
Removing candidate options improves InstructBLIP-13B performance, whereas the effect is relatively minor for GeminiProVision; the authors relate this difference to model robustness to distractions.
C.3 Effect of Introduced Options
- The authors attribute the options’ distracting effect on InstructBLIP-13B to under-confidence and suggest that models over 175B parameters are more robust due to greater capacity and sophistication.Source: §C.3 · p. 14
- 2.4% to 3.4% performance improvement followed option removal for InstructBLIP-13B, while the effect was relatively minor for GeminiProVision.Source: Figure 11 · p. 14, Figure 12 · p. 14, §C.3 · p. 14