PaperlayerView exact revision on arXiv

Source-linked AI summary

FORUM: Frozen Outputs Reconciled Using Model Agreement for Visual Grounding

Taiyo Sato, Takamasa Sanda, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama, Shunya Nagashima

arXiv:2609.37488v1cs.CVcs.CLcs.LG

TL;DR

Adversarial referring expressions with same-category distractors and negation can make even large frozen MLLMs confidently wrong, while resampling may repeat the error. FORUM fuses different frozen models using agreement-based selection and medoid localization, without fine-tuning. It exceeds the 397B-parameter reference on Ref-Adv-s, and the paper reports scope and cost limitations.

  • Problem

    On adversarial referring-expression benchmarks, same-category distractors and negation challenge frozen MLLMs, whose single-call errors can persist under resampling.

    Source: §1 · pp. 1–2

  • Method

    FORUM fuses boxes from different frozen MLLMs by selecting the region supported by the most distinct models and returning its medoid, without fine-tuning.

    Source: §1 · p. 2, §4.2 · p. 7

  • Results

    55.4 mean for FORUM's dominant lineup on Ref-Adv-s, versus 52.6 mean for the 397B Qwen3.5-A17B reference.

    Source: §5.3 · p. 10

  • Takeaways & Limitations

    Cross-model agreement and medoid localization provide a training-free fusion approach that can outperform a large single-model reference on adversarial REC.

    Source: §1 · p. 3, §6 · p. 14

    Fusion multiplies inference cost, may only match a dominant member, and is evaluated on two REC benchmarks with fixed lineups.

    Source: §6 · p. 14

Abstract

from arXiv · show

Frozen multimodal large language models (MLLMs) now solve standard referring expression comprehension with a single prompted call, yet on adversarial benchmarks with same-category distractors and negation, even the largest models are confidently wrong, and resampling repeats the error. Models built from different data and architectures rarely fall for the same confounder, so their agreement is a strong label-free signal of the correct target. We present FORUM, a training-free test-time fusion of frozen MLLMs guided by two fixed geometric rules: agreement-based selection keeps the region supported by the most distinct models, and medoid localization returns an actual member box instead of a coordinate average, so one loose prediction cannot shift the answer. Fusing three open MLLMs, FORUM surpasses the 397B-parameter published reference by a relative 5% in mean accuracy on the adversarial Ref-Adv-s benchmark, and a plain averaging ensemble by 15%. The gains transfer to standard RefCOCO+, and a balanced lineup with no dominant member still surpasses the 397B model by 5%.

1 Introduction

FORUM uses agreement among different frozen MLLMs to address adversarial referring expressions that can trap a single model. Its training-free fusion combines agreement-based region selection with medoid localization, and the paper reports gains over a 397B-parameter reference on Ref-Adv-s.

Introduction

  • Referring expression comprehension localizes an object described by natural language, while Ref-Adv adds hard same-category distractors that require reasoning about attributes, relations, and negation.Standard REC benchmarks can permit strong prompted-model performance, but the adversarial setting tests more demanding distinctions.Source: §1 · p. 1
  • A single greedy call can confidently select a look-alike distractor, and resampling that model tends to repeat its error.Prior approaches often change model weights, architecture, or the standard prompted-MLLM interface.Source: §1 · p. 2
  • Different model families often make weakly correlated errors, making their agreement a label-free signal that averaging predicted boxes does not provide.A loose or confidently wrong box can pull an averaged prediction off target, especially at strict IoU.Source: §1 · p. 2
  • Three frozen MLLMs with FORUM surpass the 397B-parameter published reference on Ref-Adv-s and improve over the strongest single member on RefCOCO+.The method combines cross-model agreement with medoid localization rather than averaging predicted coordinates.Source: §1 · pp. 2–3

2 Related Work

Related work spans referring-expression grounding, prompted MLLMs, and test-time ensembling. FORUM studies a different fusion signal and output rule: selection by model identity without confidence scores, followed by an actual member box rather than a fitted average.

2.1 Referring Expression Comprehension

  • REC methods evolved from detector-and-expression-encoder systems to end-to-end vision-language detection, while the target remains defined by context, attributes, and relations as well as category.Ref-Adv tests visual reasoning with same-category distractors and linguistic facets such as negation; Rex-Thinker instead uses candidate-wise reasoning and verification.Source: §2.1 · p. 3

2.2 Multimodal Large Language Models for Visual Grounding

  • Frozen MLLMs can be prompted to output bounding boxes directly, but precise spatial perception remains difficult and many proposed remedies require specialized training or model intervention.Grounding-capable models include Qwen-VL [1], Kosmos-2, Shikra [5], and Ferret.Source: §2.2 · pp. 3–4

2.3 Test-Time Fusion and Ensembling

  • FORUM selects by distinct-model agreement without calibrated confidence and returns an actual member box, unlike detector fusion that weights candidates or methods that fit synthetic boxes.Its prompted-MLLM interface requires no training and studies when weakly correlated errors outperform single-model scaling and plain averaging.Source: §2.3 · p. 4

3 Problem Formulation

The paper evaluates single-target referring-expression comprehension on adversarial Ref-Adv-s. It specifies the bounding-box task and IoU-based accuracy, then describes the benchmark and its evaluation protocol.

3 Problem Formulation

  • The study focuses on referring expression comprehension on the adversarial Ref-Adv benchmark.Each instance requires a single axis-aligned box for the object uniquely described by an image and expression.Source: §3 · p. 4, §3.1 · p. 5

3.1 Task and Metric

  • REC predicts one axis-aligned bounding box for the uniquely described object in an image-expression pair.The ground-truth box is denoted b* in the task formulation.Source: §3.1 · p. 5
  • A prediction is correct at threshold τ when its IoU with the ground-truth box is at least τ; invalid boxes receive zero accuracy under the official scorer.IoU is the intersection area divided by the union area.Source: §3.1 · p. 5

3.2 The Ref-Adv Benchmark

  • Ref-Adv pairs complex expressions with same-category distractors that fail the expression's linguistic constraints, reducing shortcuts such as relying only on category.The benchmark includes long expressions, multiple distractors, and negation.Source: §3.2 · p. 5
  • The evaluation uses Ref-Adv-s, a 1,142-case evaluation-only split with 40% negated expressions and released scoring code and reference predictions.Its cases are drawn from COCO val2017 and OpenImages.Source: §3.2 · p. 5

3.3 Evaluation Protocol

  • The authors use the unmodified official Ref-Adv harness on the full Ref-Adv-s split and report mean accuracy across Acc@0.5, Acc@0.75, and Acc@0.9.They also report threshold-specific accuracy and analyze results by distractor count and negation.Source: §3.3 · p. 5
  • The evaluation is limited to single-target REC, with segmentation, multi-box prediction, and video grounding out of scope.Each instance has exactly one ground-truth box.Source: §3.3 · p. 5

4 Method

FORUM fuses boxes from frozen MLLMs using two fixed rules: select the region supported by the most distinct models, then return an actual member box from that region. The design avoids training, confidence calibration, extra model calls, and coordinate averaging.

4 Method

  • FORUM queries frozen MLLMs from different families and applies agreement-based region selection followed by medoid localization, without fine-tuning or coordinate averaging.The pipeline pools candidate boxes to produce its output box.Source: §4 · p. 5

4.1 Overview

  • On adversarial inputs, errors from different model families are often weakly correlated, so cross-model agreement can help where single-model resampling repeats a distractor error.Stage A keeps the cluster supported by the largest number of distinct models; Stage B returns an actual member box rather than an average.Source: §4.1 · p. 6, Figure 2 · p. 6
  • The method uses only member output boxes and one IoU threshold, adds no model calls beyond K greedy forwards, and requires neither weight or logit access nor calibrated confidence.The design permits prompted hosted or open models to serve as members.Source: §4.1 · p. 6

4.2 Cross-Model Agreement Selection

  • Candidate boxes are linked into IoU-based clusters using a permissive threshold, and the selected cluster maximizes distinct-model support, with cluster size breaking ties.The permissive threshold accommodates scale differences; deterministic tie-breaking and distinct-model counting prevent duplicate candidates from outvoting other members.Source: §4.2 · p. 7

4.3 Medoid Localization

  • The medoid is the selected cluster's member box with the greatest overlap with the other boxes, avoiding a synthetic coordinate average.This prevents an imprecise spatial outlier from continuously dragging the output coordinates off target; medoid ties go to the earliest box.Source: §4.3 · p. 7, §4.2 · p. 7
  • Averaging and inverse-variance weighting produce synthetic boxes, while the medoid requires no precision estimates and returns an existing member prediction.The paper motivates this choice for heterogeneous localization precision, where a loose member can drag an average off target.Source: §4.3 · pp. 7–8

5 Experiments

The experiments test training-free fusion on adversarial Ref-Adv-s and standard RefCOCO+, alongside analyses of agreement, failure modes, and localization rules. FORUM improves over single-model and averaging baselines, with gains also observed for a balanced lineup.

5.1 Experimental Setup

  • FORUM uses three off-the-shelf MLLMs from distinct families in dominant and balanced lineups, with greedy primary boxes and fixed fusion rules; no test-set calibration is used.The balanced members’ single-model means span 1.6 points; the dominant lineup includes Qwen3.6, its strongest member.Source: §5.1 · p. 8

5.3 Main Results on Ref-Adv-s

  • 55.4 mean for the dominant lineup and 55.3 for the balanced lineup exceed the 397B Qwen3.5-A17B reference (52.6 mean); the dominant lineup reaches 37.2 Acc@0.9.Paired bootstrap gains over the best member and plain ensemble are +3.2 and +7.2 mean, respectively (both p < 0.001).Source: Table 1 · p. 9, §5.3 · p. 10

5.2 Implementation Details

  • Each instance uses three greedy model forwards, with optional sampling passes; the 397B reference uses fewer FLOPs, while FORUM’s largest member fits on one GPU.The authors therefore make no efficiency claim.Source: §5.2 · p. 9

5.4 Transfer to Standard REC

  • On RefCOCO+, FORUM is best on val (76.4 mean) and testB (69.2), and within 0.1 of Qwen3.6 on testA (80.3 vs. 80.4).It beats the plain ensemble on every split and exceeds Qwen3.6 at Acc@0.5 on val/testA/testB: 89.0/92.5/81.5 vs. 87.8/91.8/79.3.Source: Table 2 · p. 10, §5.4 · p. 10

5.5 Qualitative Results and Failure Analysis

  • In successful cases, cross-family agreement selects the target over plausible but linguistically inconsistent same-category distractors.Among Acc@0.5 failures, 72% have no member box on the target; the other 28% contain a correct member box that agreement does not select.Source: Figure 3 · p. 11, §5.5 · p. 11, Figure 4 · p. 11

5.6 Agreement Analysis and Facets

  • FORUM improves over the best single member and plain ensemble on every Ref-Adv-s facet, with its largest single-model gain on negated expressions (+3.7 mean).The gain narrows on scenes with seven or more distractors.Source: Table 3 · p. 12, §5.6 · p. 13
  • Cross-family error correlation is 0.47 versus 0.63 within-family, and accuracy rises with agreeing members from 31% to 60% to 85%.Across all size-K subsets of an eight-model pool, mean accuracy rises monotonically with K.Source: §5.6 · pp. 12–13

5.7 Ablations

  • On the dominant lineup, the medoid reaches 37.2 Acc@0.9 versus 16.7 for averaging; unlike averaging or coordinate-wise median, it returns an actual member box.On the balanced lineup, the medoid gains +0.8 mean and +2.7 Acc@0.9 over averaging; the Qwen-free Gemma-4 + GLM-4.6V pair reaches 53.2, above the 397B reference.Source: Table 4 · p. 13, §5.7 · pp. 13–14

6 Conclusion

FORUM uses two fixed rules to fuse frozen MLLMs, with agreement-based selection and medoid localization supporting reliable grounding on adversarial REC. The paper reports gains on Ref-Adv-s and RefCOCO+, while noting constraints on fusion cost and evaluation scope.

6 Conclusion

  • 55.4 mean on Ref-Adv-s, above every published single model up to the 397B reference; the balanced lineup reaches 55.3 and transfers gains to RefCOCO+.The conclusion attributes these results to training-free test-time fusion using agreement and medoid localization.Source: §6 · p. 14
  • Fusion multiplies inference cost, may only match a dominant model, and is evaluated on two REC benchmarks with fixed lineups.The paper suggests reporting the strongest member alongside fusion.Source: §6 · p. 14
  • Adaptive fusion, distillation into a compact model, and extensions to segmentation or video are proposed as future directions.Source: §6 · p. 14

Supplementary Material

The supplementary analyses test sensitivity to the clustering threshold and compare cross-model agreement with a model’s own confidence. Results indicate limited threshold sensitivity and show that confident predictions contradicted by other members are often wrong.

A Sensitivity to the Clustering Threshold

  • Across τ ∈[0.3, 0.7], mean accuracy varies by at most 0.4 on the dominant lineup, 0.1 on the balanced lineup, and 1.0 on RefCOCO+ testB.This supports that the single fusion hyperparameter is not tuned specifically to τ=0.5.Source: A Sensitivity to the Clustering… · p. 17

B Agreement versus Confidence

  • When the Qwen member is confident but other members disagree, accuracy is 33.3%, versus 85.0% when both confidence and agreement hold.Both confidence and agreement carry information about correctness, but fail on different instances.Source: B Agreement versus Confidence · p. 17

C Statistical Testing

Paired bootstrap tests find statistically significant Ref-Adv-s gains over the best member and plain ensemble for both lineups. Balanced-lineup subset results show that all member pairs beat individual models, and the full trio leads across metrics.

C Statistical Testing

  • The dominant lineup exceeds its best member by +3.2 and the plain ensemble by +7.2 mean; the balanced lineup margins are +5.4 and +0.8, respectively.All reported 95% confidence intervals exclude zero; the balanced lineup also gains +2.7 at strict Acc@0.9.Source: C Statistical Testing · p. 17

D Member Subsets of the Balanced Lineup

E Full Ref-Adv-s Comparison

  • The full Ref-Adv-s comparison includes additional single-MLLM baselines; the published Qwen3.5-27B leaderboard entry is 51.6.The listed fusion-member rows are greedy runs under the official harness.Source: E Full Ref-Adv-s Comparison · p. 18

F Additional Qualitative Results

Additional qualitative examples cover successful Ref-Adv-s grounding and two failure modes: selection failure and shared-distractor coverage failure.

F Additional Qualitative Results

G Cross-Family Controls

On Ref-Adv-s, cross-family fusion outperforms same-family fusion and repeated predictions from the strongest member. The controls attribute the gain to family diversity rather than single-model strength or prediction count.

G Cross-Family Controls

  • Cross-family fusion rises 5.4 points from its individual mean of 49.2, versus a 2.5-point rise for the Qwen trio from 50.0.Mean pairwise error correlations are 0.47 for cross-family members and 0.62 for the same-family trio.Source: G Cross-Family Controls · p. 22, Table 9 · p. 22
  • The controls indicate that family diversity, rather than single-model strength or prediction budget, explains the fusion gain.The balanced lineup reaches 54.6 despite a lower individual mean than the same-family trio.Source: G Cross-Family Controls · p. 22
  • 54.6 accuracy for cross-family fusion exceeds the same-family Qwen trio at 52.5 and three samples of the strongest member at 50.4.The comparison uses one primary box per member and the fixed distinct-model-count and medoid rules at τ=0.5.Source: Table 9 · p. 22, G Cross-Family Controls · p. 22

H Inference Cost

Inference-cost comparisons show trade-offs across FLOPs, memory, and latency, rather than an overall efficiency advantage. The authors frame the main comparison as one of accuracy.

H Inference Cost

  • The authors make no efficiency claim: the rerun 397B reference scores 53.7 mean, 1.7 below FORUM, so the main-paper comparison is accuracy.The 397B published mean is 52.6; the rerun and published values are reported separately.Source: H Inference Cost · p. 23, Table 10 · p. 23
  • The 397B reference uses 19.6 TFLOPs per instance, compared with 29.7 for the dominant lineup.The reported FLOPs favor the reference by 1.5×.Source: Table 10 · p. 23, H Inference Cost · p. 23
  • FORUM’s largest member fits one GPU at 67 GiB, while the 397B checkpoint requires 378 GiB; three GPUs run FORUM in 1.4 seconds versus 1.3 seconds for the reference on eight.On one GPU, the dominant lineup runs in 2.9 seconds, while the 397B reference cannot run.Source: Table 10 · p. 23, H Inference Cost · p. 23

Related papers

Loading 2609.37488v1…