PaperlayerView exact revision on arXiv

Source-linked AI summary

CUSP: Decomposable Collective Uncertainty for Multi-Agent Multimodal Reasoning

Chung-En Johnny Yu, David Garcia, Brian Jalaian, Nathaniel D. Bastian

arXiv:2609.05708v1cs.AIcs.LGcs.MA

TL;DR

System-level reliability remains difficult to measure when heterogeneous VLMs are aggregated because individual or aggregate confidence may not reflect ensemble behavior. CUSP maps sampled VLM responses into a shared semantic space, pools their opinions, and reports collective uncertainty and JSD. Across static ensembles and a multi-agent trajectory, these signals support error detection, abstention, and conflict diagnosis, subject to semantic-clustering, sampling, and evaluation-scope limitations.

  • Problem

    Heterogeneous VLM aggregation can improve reasoning, but individual or aggregated confidence does not measure system-level reliability.

  • Method

    CUSP is a training-free framework that maps sampled VLM responses into a shared semantic space, pools model opinions, and reports collective uncertainty and JSD without logits or calibration labels.

  • Results

    CUSP supports prediction-error detection, abstention, and conflict diagnosis across static ensembles and a multi-step multi-agent trajectory.

  • Takeaways & Limitations

    Collective uncertainty captures pooled dispersion, while JSD separates model conflict through an exact entropy decomposition.

  • Takeaways & Limitations

    CUSP depends on semantic-clustering granularity and judge errors, while repeated sampling overhead grows with the number of models and reasoning rounds.

Abstract

from arXiv · show

Aggregating heterogeneous vision-language models (VLMs) can improve multimodal reasoning, but neither an individual model's confidence nor that of the aggregated answer measures reliability at the system level. We present CUSP (Collective Uncertainty through Semantic Opinion Pooling), a training-free uncertainty quantification framework that maps multiple VLM responses to a shared semantic response space, pools them into a pooled semantic opinion, and reports two complementary system-level signals: collective uncertainty, the dispersion of the pooled opinion, and Jensen-Shannon divergence (JSD), the conflict among the model-level opinions. Within this pooled semantic opinion, the unnormalized collective entropy decomposes exactly into the mean of the models' individual semantic entropies and the JSD, separating total dispersion from model conflict. Requiring neither token logits nor calibration labels, CUSP applies to open-weight and commercial VLMs alike. In static multi-VLM ensembles, collective uncertainty is the strongest signal in the small-model regime (0.764 AUROC for prediction-error detection, 0.889 AUARC for abstention), outperforming uncertainty baselines majority voting and naive selection by 4.7 to 15.8 points and widening its margin as the ensemble grows; JSD is strongest in the evaluated commercial regime (0.819 AUROC, 0.910 AUARC) and ranks hard-answer model conflict with AUROC up to 0.982. The pooled prediction also improves accuracy over the average single model by 5.6 to 13.0 points. Over the full trajectory of a multi-step, multi-agent system, subagent collective uncertainty ranks system failures above chance (0.619 AUROC) and gives the best abstention ordering among the evaluated signals (0.699 AUARC).

1 Introduction

CUSP addresses the lack of system-level reliability measures for heterogeneous VLM ensembles by mapping responses into a shared semantic space and separating pooled dispersion from model conflict. It evaluates these signals across static ensembles and multi-step multi-agent trajectories.

  • Motivation: Individual confidence and aggregated-answer confidence do not characterize reliability for systems combining heterogeneous VLMs.Existing methods typically model one response or fail to separate total ensemble uncertainty from conflict.
  • CUSP: CUSP maps sampled free-form responses into a shared semantic space, pools model-level opinions, and reports collective uncertainty and JSD without logits or calibration labels.Collective uncertainty measures pooled-opinion dispersion, while JSD measures conflict among model opinions.
  • CUSP: The unnormalized pooled entropy decomposes exactly into mean individual semantic entropy plus JSD, separating average model uncertainty from model conflict.Thus, similar total uncertainty can arise from uniform hesitation or confident disagreement.
  • Evaluation: CUSP evaluates collective uncertainty and JSD in static multi-VLM ensembles and a multi-step, multi-agent evidence-acquisition trajectory.The evaluation broadens validation beyond a single response without claiming that CUSP controls the trajectory.
  • Related work: Prior work covers individual-model uncertainty, ensemble aggregation, and single-agent trajectories but does not jointly characterize heterogeneous multi-model system reliability.The related-work gap spans semantic alignment, conflict, and trajectory-level risk across concurrent agents.

3 Method

CUSP converts sampled VLM responses into aligned semantic opinions, pools them uniformly, and selects the most probable semantic class as the system prediction. It quantifies pooled dispersion and decomposes it exactly into mean individual uncertainty and JSD-based conflict.

  • Shared semantic response space: Each VLM generates sampled free-form responses that are mapped into a shared semantic response space.An LLM-based semantic-equivalence judge groups responses expressing the same meaning into discrete classes.
  • Model opinions: Each model’s empirical semantic opinion assigns class probabilities by relative frequencies of its sampled responses.All models use the same finite support, with zero mass for classes absent from a model’s samples.
  • Opinion pooling: CUSP uses uniform linear opinion pooling so every model-level opinion has equal influence.The pooled opinion supports both response selection and uncertainty quantification.
  • Collective uncertainty: The system returns a representative response from the semantic class with the largest pooled probability and computes collective entropy from the pooled opinion.Normalized collective uncertainty places scores on a common scale across inputs with different numbers of semantic classes.
  • Decomposing collective uncertainty with JSD: The exact decomposition expresses pooled entropy as mean individual entropy plus JSD, with JSD quantifying differences among semantic probability assignments.JSD is nonnegative and equals zero exactly when all models assign identical mass to every semantic class.
  • Decomposing collective uncertainty with JSD: Collective uncertainty captures total pooled dispersion, whereas JSD isolates conflict among model opinions and does not imply correctness when models agree on an incorrect response.The same definitions apply round by round when VLMs operate as subagents in a multi-agent system.

4 Experiments

Experiments evaluate CUSP’s uncertainty signals, pooled predictions, and conflict diagnosis across static heterogeneous VLM ensembles and a multi-step multi-agent system. Collective uncertainty generally supports prediction-error detection and abstention, while JSD is particularly useful for model-conflict diagnosis.

  • Static multi-VLM ensembles: 0.764 AUROC and 0.889 AUARC make collective uncertainty strongest in the small-model regime for prediction-error detection and abstention.In the large-model regime, it also leads with 0.906 AUROC and 0.987 AUARC.
  • Static multi-VLM ensembles: 0.819 AUROC and 0.910 AUARC make JSD strongest in the commercial regime, outperforming both baselines on both metrics.Collective uncertainty trails Majority Voting by 0.5 AUROC points but exceeds it by 3.6 AUARC points.
  • Ensemble-size analysis: 0.840 AUROC is collective uncertainty’s small-model result at larger ensemble size, rising from 0.757 while Majority Voting falls below it from three models onward.Naive Selection declines from 0.660 to 0.519 AUROC, whereas JSD declines from 0.709 to 0.588 because it targets model conflict rather than prediction error.
  • Aggregated accuracy and latency: 13.01, 5.56, and 8.70 percentage points are CUSP’s pooled-prediction accuracy gains over the average individual VLM in small, large, and commercial regimes.The evaluation also reports aggregated prediction accuracy across all systems in each model regime.
  • Model-conflict diagnosis: Conflict-AUROC measures how well collective uncertainty and JSD rank questions where at least one model selects a different top semantic response class.Table 2 reports this hard-answer model-conflict discrimination across model regimes.
  • Multi-agent trajectory evaluation: 0.699 Traj-AUARC is collective uncertainty’s best trajectory-level abstention result under temporal-maximum aggregation.It is best under both temporal mean and maximum summaries and leads in three of four mean/max comparisons; mean Round-AUROC is 0.594.

5 Discussion

Uniform pooling is a conservative default that separates reliability measurement from downstream trust policies, but CUSP remains bounded by task, sampling, mapping, and evaluation-scope limitations.

  • Pooling policy: Lower entropy indicates concentration rather than correctness, so confidence-weighted pooling can over-weight a miscalibrated, incorrect model.Uniform pooling avoids this calibration assumption and preserves equal influence among model opinions.
  • Pooling policy: Uniform pooling does not guarantee optimality for every downstream task; it separates collective reliability measurement from policies that assign unequal trust.The paper presents uniform pooling as a conservative default rather than a universally optimal policy.
  • Operational scope: Trajectory scores support retrospective ranking of completed predictions for rejection or review, not online stopping, re-query, or abstention policies.The trajectory analysis does not establish online control of the multi-agent process.
  • Limitations: CUSP depends on semantic-clustering granularity, judge errors, ten-sample opinion estimates, and sampling costs that grow with models and reasoning rounds.Free-form multi-agent settings require an LLM-based judge, and their semantic-mapping costs may be non-negligible.
  • Limitations: The commercial evaluation covers 20 three-VLM systems drawn from six selected models and is not a population estimate for commercial VLMs.The signals also miss shared misconceptions and incorrect orchestration of consistent evidence, so they are complementary monitoring signals rather than correctness certificates.

6 Conclusion

CUSP pools sampled free-form VLM responses in a shared semantic space and exactly decomposes collective entropy into individual uncertainty and model conflict. Across static and multi-agent settings, its signals support error detection, abstention, and conflict diagnosis, with collective uncertainty strongest for trajectory-level abstention ordering.

  • Conclusion: CUSP pools sampled free-form VLM responses in a shared semantic space, where collective entropy decomposes into mean individual uncertainty and JSD-based model conflict.The framework is training-free and produces two complementary system-level signals.
  • Conclusion: Across static ensembles, collective uncertainty supports prediction-error detection and abstention, while JSD diagnoses conflict and is the strongest commercial-regime error signal.These evaluations cover static model regimes and ensemble sizes.
  • Conclusion: In the multi-agent setting, subagent signals remain above chance, with collective uncertainty providing the best trajectory-level abstention ordering.The conclusion reports retrospective trajectory-level signal performance without claiming trajectory control.
  • Conclusion: Future work should reduce sampling and semantic-mapping costs and test whether the signals support online verification or evidence-acquisition policies.These are identified directions rather than established capabilities.

S1 Additional Notation

The notation indexes each evaluation question–image pair and represents each model’s empirical opinion over a shared semantic response space.

  • Notation: For each question–image pair x_i, index i identifies one evaluation item, and p_i,k denotes model M_k’s empirical opinion evaluated on x_i.The notation uses item-indexed shorthand for model-level opinions.
  • Notation: Each item’s semantic opinion is supported on the shared response space S_xi = {s_1, . . . , s_Ji}.The support consists of semantic response classes for that evaluation item.
  • Notation: CUSP selects the top semantic response class and reports normalized collective uncertainty U_M,i and normalized JSD score D_M,i.These are item-indexed counterparts of the corresponding main-paper signals.

S2 CUSP Semantic Opinion Pooling Algorithm

Algorithm S.1 summarizes the complete CUSP procedure described in the paper’s Section 3.

  • Algorithm: Algorithm S.1 summarizes the complete CUSP procedure described in Section 3.The supplied passage identifies the algorithm as the procedural summary.

S3 Baseline Definitions

The baselines compare hard-vote aggregation, lowest-entropy model selection, and average individual-model performance against system-level uncertainty measures.

  • Majority Voting: Majority Voting assigns one vote per VLM’s most probable semantic response and uses normalized vote-distribution entropy as uncertainty.
  • Tie handling: Ties are resolved deterministically by the first class or model in the configured benchmark or combination order, requiring no additional model call.
  • Naive Selection: Naive Selection chooses the model with the lowest individual semantic entropy and uses that model’s prediction and normalized entropy.
  • Average individual reference: The average-individual reference averages component VLM accuracies or uncertainty scores and does not produce an ensemble prediction.

S4 Evaluation Metrics

The evaluation defines ranking metrics for prediction-error detection, conflict diagnosis, abstention quality, and latency, while distinguishing aggregation overhead from VLM sampling time.

  • AUROC: AUROC measures how well a ranking score orders incorrect versus correct predictions, with 0.5 representing chance ranking.
  • Conflict-AUROC: Conflict-AUROC uses hard-answer model conflict as its positive label rather than prediction error, so it measures conflict ranking.
  • AUARC: AUARC summarizes retained accuracy after rejecting increasingly uncertain predictions, but it is an offline selective-reliability metric rather than evidence of an online abstention policy.
  • Latency: VLM UQ sampling latency sums component-model inference times and reports their per-system p50, while aggregation overhead is measured separately.
  • Latency caveat: Sub-microsecond ordering differences should not be interpreted as practically meaningful because aggregation benchmarks can vary slightly across fresh runs.
  • Latency: Sampling latency rises from 2.04 seconds for two VLMs to 8.30 seconds for eight, whereas CUSP’s mean aggregation overhead is 3.97 microseconds.

S5 Multi-Agent Trajectory Metrics

Trajectory metrics summarize uncertainty across rounds and evaluate both round-level probe-error ranking and completed-trajectory failure or abstention ordering.

  • Round-wise metrics: Round-level signals use per-round semantic uncertainty, vote entropy, and vote conflict to assess non-interventional probe-answer errors.
  • Trajectory metrics: Trajectory summaries aggregate round scores over completed systems, with Traj-AUROC using final orchestrator failures and Traj-AUARC ranking rejected trajectories.
  • Round-wise results: 0.594 mean Round-AUROC makes collective uncertainty the strongest average signal across rounds, ahead of MeanSE at 0.584 and JSD at 0.580.
  • Round-wise results: Round-AUROC generally decreases as targeted evidence resolves visual ambiguity and leaves errors driven by integration, reasoning, or shared subagent mistakes.

S6 Pooling-Weight Ablation

The ablation compares uncertainty-weighted pooling with CUSP’s uniform pooling and evaluates how pooling choices affect static and trajectory-level uncertainty quality.

  • Pooling rules: Uncertainty-weighted pooling assigns greater weight to models with lower individual semantic entropy, with a limiting convention for zero-entropy models.
  • Pooling rules: The implementations use 10^-12 and 10^-8 only as numerical safeguards approximating the zero-entropy limit, not as formal pooling parameters.
  • Trajectory ablation: Uniform pooling improves trajectory Traj-AUROC and Traj-AUARC over uncertainty weighting by 0.009 and 0.006 under temporal mean, and by 0.019 and 0.011 under temporal maximum.
  • Trajectory ablation: The ablation supports uniform pooling for the evaluated trajectory but does not establish that uncertainty weighting is universally inferior.

S7 Implementation Details and Prompts

The appendix documents the model, hardware, sampling, pooling, multi-agent architecture, ablations, and prompt templates used to implement and evaluate CUSP.

  • S7.1 Multi-VLM models and response mapping: The static experiments use Ministral-3, Granite Vision, Qwen2.5-VL, Qwen3-VL, Qwen3.5, Qwen3.6, Gemma3, Gemma4, LLaVA-NeXT, and LLaVA-Phi3.These identifiers correspond to the final analysis configurations.
  • S7.1 Multi-VLM models and response mapping: Local inference ran on a DSS 8440 Cauldron server with an Intel Xeon Gold 5218R CPU and eight 48-GB NVIDIA A40 GPUs, while commercial models used hosted APIs.Reported local latency measurements were collected on this hardware.
  • S7.1 Multi-VLM models and response mapping: The main response uses temperature 0.1 and ten UQ samples use temperature 1.0 where commercial APIs expose decoding control.Claude Sonnet 5 and Kimi K2.6 omit temperature because their interfaces do not expose it; N = 10 was fixed a priori.
  • S7.2 Multi-agent system architecture: Across three evidence-acquisition rounds, a captioner supplies scene context, an orchestrator instructs three heterogeneous VLM subagents, and CUSP aggregates their stochastic evidence samples.A semantic-clustering judge maps samples into a shared response space.
  • S7.2 Multi-agent system architecture: Pooling-weight ablations and aggregation-overhead ablations compare uncertainty-weighted and uniform pooling, with overhead measured above the same VLM UQ sampling latency.The overhead table reports additional ∆Lµs, with lower values indicating faster execution.
  • S7.3 Multi-agent models and generation roles: The multi-agent implementation uses Gemma4, Ministral-3, and Qwen3.5 evidence agents, with LFM2-24B for orchestration and semantic-equivalence judging.Each evidence branch produces one temperature-0.1 response and ten temperature-0.9 UQ samples.
  • S7.3 Multi-agent models and generation roles: Free-form clustering pools structured-evidence samples from successful VLM responses and requires bidirectional semantic agreement, with exact matches accepted directly and matching extracted integer sets required for explicit integer claims.Representatives come from selected clusters; uniform-pooling ties use the earliest cluster index, while uncertainty-weighted pooling first selects the lowest-entropy branch.
  • S7.3 Multi-agent models and generation roles: Static experiments use base-2 logarithms and the multi-agent runtime uses natural logarithms, but normalized quantities are base-invariant and reported unnormalized values are in bits.The appendix also provides tables for trajectory ablations, model identifiers, and prompt templates for clustering, evidence extraction, captioning, instruction generation, and final answers.
Loading 2609.05708v1…