PaperlayerView exact revision on arXiv

Source-linked AI summary

GAM-Agent: Game-Theoretic and Uncertainty-Aware Collaboration for Complex Visual Reasoning

Jusheng Zhang, Yijia Fan, Wenjun Lin, Ruiqi Chen, Haoyi Jiang, Wenhao Chai, Jian Wang, Keze Wang

arXiv:2505.23399v2cs.AI

TL;DR

Complex visual reasoning remains difficult for single-model or simple-ensemble VLMs, especially under multi-step reasoning and visual ambiguity. GAM-Agent addresses this with specialized agents that exchange structured claims, evidence, and uncertainty in an adaptive non-zero-sum debate. Across four challenging benchmarks, it consistently improves VLM performance, with especially notable gains for smaller models and continued gains for larger ones.

  • Problem

    Existing VLM approaches rely largely on single models or simple ensembles and struggle with multi-step reasoning and visual ambiguity in complex visual tasks.

  • Method

    GAM-Agent models visual reasoning as a non-zero-sum game among specialized agents that communicate structured claims, evidence, and uncertainty and debate when ambiguity or disagreement arises.

  • Results

    GAM-Agent consistently improves performance across four challenging benchmarks and VLM backbones, with gains of +5.8% to +6.7% for smaller models and +2.6% to +4.1% for larger models on MMBench_V11_Test and MVBench_Test.

  • Takeaways & Limitations

    The framework provides a modular approach for more robust and interpretable multi-agent multimodal reasoning across lightweight and strong foundation models.

Abstract

from arXiv · show

We propose GAM-Agent, a game-theoretic multi-agent framework for enhancing vision-language reasoning. Unlike prior single-agent or monolithic models, GAM-Agent formulates the reasoning process as a non-zero-sum game between base agents--each specializing in visual perception subtasks--and a critical agent that verifies logic consistency and factual correctness. Agents communicate via structured claims, evidence, and uncertainty estimates. The framework introduces an uncertainty-aware controller to dynamically adjust agent collaboration, triggering multi-round debates when disagreement or ambiguity is detected. This process yields more robust and interpretable predictions. Experiments on four challenging benchmarks--MMMU, MMBench, MVBench, and V*Bench--demonstrate that GAM-Agent significantly improves performance across various VLM backbones. Notably, GAM-Agent boosts the accuracy of small-to-mid scale models (e.g., Qwen2.5-VL-7B, InternVL3-14B) by 5--6\%, and still enhances strong models like GPT-4o by up to 2--3\%. Our approach is modular, scalable, and generalizable, offering a path toward reliable and explainable multi-agent multimodal reasoning.

1 Introduction

Existing VLMs largely rely on single models or simple ensembles, leaving multi-agent collaboration underused for complex visual reasoning. GAM-Agent addresses this gap with a non-zero-sum, uncertainty-aware collaboration framework that structures claims and evidence while enabling progressive interaction toward consensus.

  • Existing VLM approaches often struggle with multi-step reasoning and visual ambiguity because they rely on single models or simple ensemble strategies.
  • Recent VLM debate methods show promise but typically use averaging or voting rather than strategic interactions grounded in visual reasoning.
  • GAM-Agent models complex visual reasoning as a non-zero-sum game in which multiple agents collaborate to reach consensus.
  • Agents share uncertainty assessments and engage in progressive interaction that guides the system step by step toward consensus.
  • The framework uses Base Agents for distinct visual interpretations and evidence generation, while Critical Agents evaluate factual accuracy, logical coherence, and completeness.

2 Methodology

GAM-Agent combines specialized visual agents, structured claim-evidence representations, uncertainty estimation, and adaptive debate to refine visual reasoning. Its controller aggregates responses, detects uncertainty or conflict, and iterates on disputed claims until convergence, resource limits, or stagnation.

  • GAM-Agent is defined as a six-tuple containing expert agents, analysis mappings, uncertainty functions, evidence mappings, a claim parser, and a debate module.
  • Structured reasoning: The pipeline parses initial responses into claims, confidence scores, textual evidence, and visual-region references before assessing uncertainty convergence.
  • Agent roles: Base Agents generate initial visual interpretations, while Critical Agents verify factual accuracy, logical coherence, and completeness.
  • Uncertainty quantification: When generation probabilities are available, uncertainty uses token-level entropy and top-two probability differences; otherwise, semantic uncertainty markers provide the estimate.
  • Adaptive collaboration: Initial responses receive uncertainty-based weights, while weighted uncertainty and inter-expert conflict jointly determine whether iterative debate begins.
  • Iterative debate: Each debate round identifies disputed low-confidence claims, gathers arguments and confidence updates, integrates revised views, and stops at convergence, resource limits, or stagnation.

3 Experiments

Experiments evaluate GAM-Agent across image, video, and fine-grained visual reasoning benchmarks, alongside ablations of its collaboration and debate mechanisms. GAM-Agent consistently improves base VLM performance, with especially strong gains for smaller models and efficient convergence in the ablation study.

  • Experimental Setup: GAM-Agent is evaluated against five multi-agent frameworks using Qwen2.5VL, InternVL3, and GPT-4o-0513 on MMMU, MMBench_V11_Test, and MVBench_Test.The compared frameworks are DMAD, DMPL, ChatEval, MAD, and DebUnc.
  • Image and Video Understanding: +5.8% to +6.7% gains occur for smaller models on MMBench_V11_Test and MVBench_Test, including +6.61% for InternVL3-14B/MMBench.Larger models also improve by +2.6% to +4.1%, including +3.91% for GPT-4o-0513/MMBench.
  • Image and Video Understanding: 90.15% (+6.61%) is achieved by GAM-Agent on InternVL3-14B/MMBench_V11_Test, exceeding DebUnc’s 88.86% (+5.32%).The comparison reports GAM-Agent’s stronger improvement over the base model on this benchmark.
  • Fine-Grained Visual Reasoning: +5.25% overall raises Qwen2.5VL-7B to 66.51% on V*Bench, while InternVL3-14B reaches 70.33% (+4.80%) and GPT-4o-0513 reaches 78.32% (+3.60%).GAM-Agent consistently achieves the highest scores among the compared multi-agent frameworks on V*Bench.
  • Ablation Experiments: 88.80% accuracy with 1.76 average debate rounds is achieved by the full GAM-Agent ablation configuration.Removing Uncertainty or Dynamic Weights increases debate length and cost, while disabling Iterative Debate reduces accuracy by 4.28%.
  • Debate Length: 86.53% accuracy is reached at three maximum debate rounds, up from 82.97% with zero rounds, after which accuracy largely plateaus.The study varies max_debate_round from 0 to 9 using Qwen2.5VL-7B on MMBench_V11_Test.
  • Expert Weight Dynamics: Expert weights and system uncertainty are tracked across three debate rounds to show dynamic modulation during reasoning.The analysis covers Relational Reasoning, Scene Description, and OCR experts across Qwen2.5VL and InternVL3 models.

4 Related Works

Related work situates GAM-Agent at the intersection of multi-agent collaboration, uncertainty quantification, and visual reasoning. The paper distinguishes its uncertainty-aware, game-theoretic collaboration from single-model, voting, and basic ensemble approaches.

  • Multi-Agent Systems: Multi-agent systems use collaborative agents to improve complex reasoning, code generation, and knowledge integration in LLMs.Prior work includes multi-agent debate for factuality and reasoning and communicative agents for simulation.
  • Uncertainty Quantification: Uncertainty quantification supports reliable decision-making in visual reasoning, where occlusion, poor lighting, and combined visual-linguistic uncertainty create ambiguity.Prior methods include entropy-based measures and semantic analysis.
  • Visual Reasoning: Current visual reasoning strategies often rely on single models or basic ensembles that struggle with multi-step reasoning and visual ambiguities.GAM-Agent instead uses game-theoretic collaboration with uncertainty and intermediate reasoning.

5 Conclusion

The conclusion presents GAM-Agent as an uncertainty-guided, game-theoretic framework for robust and interpretable multimodal reasoning. Experiments across four benchmarks report consistent improvements across lightweight and strong VLM backbones.

  • Framework: GAM-Agent models reasoning as a non-zero-sum game among specialized base agents and a critical verification agent using structured claim-evidence interactions.The design identifies ambiguities, triggers multi-round debates, and adjusts reasoning based on confidence and disagreement.
  • Conclusion: Experiments across four challenging benchmarks consistently improve performance across lightweight and strong VLM backbones.The conclusion frames this result as a direction for self-reflective and multi-agent multimodal reasoning.

Supplementary Material Overview

The supplementary material provides additional experimental, theoretical, and case-study support for GAM-Agent. It also examines cost-performance trade-offs on MMBench_TEST_V11 across several commercial and framework-derived VLM configurations.

  • Supplementary Contents: The supplementary material includes cost-performance analyses, uncertainty evaluations, theoretical derivations, hyperparameter studies, prompt configurations, and case analyses.These materials are intended to support the framework’s robustness and limitations.
  • Cost-Performance Analysis: Cost-performance analysis on MMBench_TEST_V11 compares cost per instruction and overall accuracy for commercial VLMs and GAM-Agent configurations.The compared commercial models include Qwen2.5VL, Gemini 2.5 Pro Preview, Gemini 2.0 Pro, GPT-4o, and Claude 3.7.

A.2 Experiment Result

GAM-Agent configurations combine strong MMBench accuracy with substantially lower local inference costs, while iterative debate is evaluated as a cost-performance trade-off against unstructured self-debate.

  • 92.2% accuracy at approximately $0.00022 per instruction makes InternVL3-78B (Ours) the strongest reported cost-performance configuration.
  • 90.4% accuracy at $0.00011 per instruction makes one Qwen2.5VL-72B configuration approximately 1.36 times more cost-effective than its API counterpart.
  • The cost comparison examines cost per instruction and overall accuracy on MMBench_TEST_V11, with “Ours” using local inference cost calculations.
  • 92.2% versus 89.3% accuracy is achieved by InternVL3-78B (Ours) compared with Gemini 2.5 Pro Preview at approximately 1/3.4th of the cost.
  • The debate-cost experiment compares GAM-Agent with baseline multi-round self-debate across one, two, and three rounds using average tokens processed per instance.

B.2 Experimental Results

Across debate rounds and model scales, GAM-Agent evaluates whether structured uncertainty handling improves visual reasoning while controlling debate cost and adapting collaboration to uncertainty.

  • Debate Results: After three rounds, GAM-Agent reaches 91.90% accuracy with InternVL3-78B versus 90.20% for the single-agent baseline.
  • Debate Results: After three rounds, GAM-Agent achieves 89.02% accuracy for Qwen2.5VL-7B at $0.00009, compared with 86.50% for the baseline.
  • Debate Results: GAM-Agent’s cost increases with additional debate rounds, while structured uncertainty and conflict detection are designed to terminate debate when consensus is reached.
  • Uncertainty Handling: The extended uncertainty analysis covers Qwen2.5VL models from 3B to 72B and InternVL3 models from 2B to 78B on MMBench.
  • Uncertainty Handling: UA measures uncertainty identification, CE measures confidence-accuracy consistency, and DA measures uncertainty-responsive decision adaptation.

C.3 Experimental Results and Analysis

GAM-Agent consistently improves uncertainty management and visual reasoning performance across model sizes, while its debate termination mechanism exposes a measurable accuracy–efficiency trade-off. The framework’s non-zero-sum design supports collective improvement through collaboration, though theoretical equilibrium guarantees require simplifying assumptions.

  • Uncertainty Management: GAM-Agent outperforms baseline frameworks on uncertainty accuracy, calibration error, and dynamic adaptability across Qwen2.5VL and InternVL3 model sizes.The reported comparison includes DMAD, DMPL, ChatEval, MAD, and DebUnc.
  • Uncertainty Management: As model size increases, GAM-Agent’s uncertainty handling improves, with Qwen2.5VL UA rising from 0.72 to 0.82 and InternVL3 UA from 0.70 to 0.84.Across both series, calibration error decreases and dynamic adaptability increases with scale.
  • Performance Correlation: GAM-Agent’s stronger uncertainty metrics coincide with the highest MMBench accuracy for each base model, including 90.56% for Qwen2.5VL-72B.The passage describes this relationship as a positive correlation rather than a causal result.
  • Debate Convergence: The default termination analysis compares model accuracy, average debate rounds, and primary termination reasons across MMBench tasks.The cited figure is presented as an overview of performance and convergence dynamics under default conditions.
  • Debate Termination: Setting θU,term to 0.10 increases accuracy by approximately 0.2% to 0.4% but raises debate rounds by about 25% to 31%, whereas 0.20 reduces rounds by about 22% to 26% while lowering accuracy by approximately 0.9% to 1.2%.The threshold sensitivity analysis identifies θU,term = 0.15 as an optimal balance.
  • Game-Theoretic Analysis: The game-theoretic formulation permits collective utility gains through improved consistency and reduced system uncertainty, while mixed-strategy equilibria provide a fallback when pure-strategy guarantees are difficult.The existence of pure-strategy equilibria depends on properties such as compact convex strategy spaces and quasi-concave utilities.

E Theorem I: Fundamental Game Theoretic Framework of GAM-Agent

GAM-Agent models visual reasoning as a non-zero-sum game in which quantified uncertainty guides agent influence, collaboration, and debate. Its framework combines uncertainty assessment, evidence and claim processing, utility optimization, and dynamic interaction.

  • Uncertainty is estimated from token-generation behavior or semantic markers, capturing hesitation, weak token preference, or uncertainty expressed in responses.High entropy indicates hesitation, while a small top-token probability gap indicates weak confidence; semantic markers provide a fallback when probabilities are unavailable.
  • GAM-Agent represents multi-agent visual reasoning as a non-zero-sum game whose players generate response–uncertainty pairs for an image-question instance.The game state includes joint agent outputs and an associated influence-weight vector.
  • Agent utility combines individual confidence, semantic consistency with influential peers, and a system-wide uncertainty cost.The formulation rewards low individual uncertainty and agreement while penalizing high weighted system uncertainty.
  • Entropy regularization prevents degenerate all-influence allocation, while the resulting optimization yields analytically derived influence weights.The regularized optimization balances cooperative value against concentration of influence.
  • The framework’s claimed mathematical contribution is an end-to-end chain from uncertainty quantification through utility calculation to mechanism design.GAM-Agent presents uncertainty as a quantifiable game variable and derives system behavior from utility optimization rather than imposing it externally.

G Theorem III: Differences in Mathematical Construction between GAM-Agent and Traditional Multi-Agent Methods

GAM-Agent differs from traditional multi-agent reasoning methods by integrating uncertainty into an explicit game-theoretic optimization framework. Traditional approaches typically rely on direct aggregation, fixed rules, or heuristic confidence signals.

  • GAM-Agent’s mathematical construction explicitly integrates uncertainty, mechanism derivation, and collaborative optimization, unlike traditional methods without game-theoretic optimization.Its framework is presented as a distinct mathematical construction rather than a procedural interaction scheme.
  • Simple aggregation methods combine independently generated outputs but lack agent interaction, explicit uncertainty, and dynamic influence adjustment.Examples include majority voting, averaging, and fixed weighted averages.
  • Rule-based debates prescribe agent behavior and interaction through fixed roles and rules rather than deriving them from current agent states through optimization.The interaction policy is controlled by preset rules and history instead of an explicit utility-maximization process.
  • Traditional aggregation often omits uncertainty entirely, while rule-based debates usually express confidence qualitatively or heuristically.GAM-Agent instead converts uncertainty into mathematical effects on weighting and debate through a unified framework.

D.4.2 Mathematical Basis of Interaction Mechanisms and Influence Allocation: GAM-Agent: Influence weights w∗

GAM-Agent allocates influence through continuous, dynamic weights derived by maximizing a regularized collaborative utility. This contrasts with traditional methods that use equal, binary, or rule-assigned influence.

  • GAM-Agent derives influence weights by optimizing regularized total utility rather than assigning them through preset rules.The optimization objective explicitly includes uncertainty and collaboration terms.
  • Its weights are continuous and dynamic, changing with agent uncertainty and semantic consistency with other agents.The resulting allocation reflects relative agent value instead of a fixed participation rule.
  • Traditional methods may give agents equal influence, use binary voting, or confer influence through a preset judge role.These mechanisms do not derive influence continuously from uncertainty and inter-agent consistency.
  • The utility objective quantifies individual certainty, mutual consistency, and system uncertainty as explicit collaborative dimensions.It defines mathematical trade-offs among these dimensions and links utility maximization to high-quality consensus.
  • GAM-Agent is presented as more mechanistically explicit than traditional methods because its components and influence mechanism trace back to an uncertainty-based model.The authors connect this explicitness to more flexible, adaptive, and interpretable visual reasoning.

H Hyperparameter Ablation

Hyperparameter ablations show that GAM-Agent’s performance depends on balancing uncertainty sensitivity, debate triggering, and uncertainty composition. The reported default configuration is presented as a practical trade-off for MMBench.

  • Debate triggers: Lower debate thresholds increase trigger rates, debate rounds, and cost, sometimes producing marginal accuracy gains that are not always cost-effective.The combined θU = 0.35 and θC = 0.45 setting reaches 88.98% with an 82% trigger rate and higher cost.
  • Debate triggers: Higher debate thresholds reduce debate frequency, rounds, and cost but can lower accuracy when critical conflicts are overlooked.The reported accuracies are 87.93% for θU = 0.65 and 88.10% for θC = 0.75.
  • Uncertainty composition: Approximately equal weighting of entropy and top-probability difference performs best for generation-process uncertainty.The default αΦ = 0.5 and βΦ = 0.5 configuration achieves 88.80%.
  • Base configuration: 88.80% accuracy is achieved by the default InternVL3-14B configuration on MMBench_TEST_V11 with N = 3 and Kmax = 3.This configuration serves as the baseline for component and hyperparameter ablations.

K.2 Unsuccessful Case

The unsuccessful cases reveal that GAM-Agent can produce incorrect judgments when experts rely on superficial cues, misinterpret ambiguous concepts, or fail to align with the question’s semantics. Critic modules and aggregation may not correct confidently wrong, semantically misgrounded reasoning.

  • Confidence and aggregation: High-confidence expert outputs can still yield an incorrect final aggregation.The case shows that low uncertainty among all three analysis experts did not guarantee prediction accuracy.
  • Semantic misalignment: Ambiguous concepts such as “colorful” can produce divergent interpretations when experts lack shared semantic grounding.Experts emphasized saturation rather than color diversity, which the passage identifies as more appropriate for the query.
  • Expert selection: The framework may activate irrelevant experts, such as Text/OCR analysis when an image contains no text, diluting pertinent evidence during aggregation.The existing mechanism cannot dynamically suppress irrelevant expert responses.
  • Critic limitations: Critic agents may fail to identify and correct confidently wrong interpretations that are semantically misgrounded.The Fact, Completeness, and Logic Checkers did not effectively intervene in the reported failure case.
  • Systemic errors: Coordinated incorrect reasoning can amplify systemic bias when higher-level semantic validation or external verification is absent.The passage specifically gives feature-level consistency checks as an example of missing external verification.
  • Visual cue limitations: Experts can overrely on superficial visual cues, overlooking lighting, shadows, texture, and environmental influences in color judgments.The rhinoceros case attributes perceptual discrepancies to illumination shifts and related contextual factors.
  • Query alignment: Expert reasoning may focus on contextual details rather than the semantic intent of the question, producing conclusions misaligned with the query.The Scene Description expert emphasized the scene’s tranquil atmosphere instead of directly addressing whether the rhinoceroses had the same color.
  • Critic limitations: Critique modules may overlook both omitted visual factors and deviations from the core question.The Completeness Checker did not emphasize illumination and texture, while the Logic Checker did not flag the reasoning’s deviation from the query.
Loading 2505.23399v2…