GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation

1. Introduction

• Earlier benchmarks focus on simple phrases or relations, while longer descriptions can still permit keyword shortcuts and do not test rejection of mismatched details.Existing benchmarks already yield over 90% accuracy on RefCOCO and nearly 90% on Ref-L4.Source: §1 · p. 2
• GroundingME comprises 1,005 samples organized around Discriminative, Spatial, Limited, and Rejection challenges, with a finer twelve-subcategory hierarchy.The dimensions target subtle object differences, complex relations, limited visual features, and misleading descriptions.Source: §1 · p. 2
• Qwen3-VL-235B-A22B achieves only 45.1% accuracy, and most evaluated models score 0% on Rejection.The reported failure persists as model scale increases.Source: §1 · p. 2
• Test-time scaling selects among thinking trajectories, while data-mixture training with negative samples raises Qwen3-VL-8B-Instruct’s Rejection accuracy from 0% to 27.9%.The paper also reports that test-time scaling improves performance across subtasks, particularly reasoning-intensive categories.Source: §1 · p. 2

Multimodal Large Language Models

• Existing grounding models perform strongly on prior benchmarks, motivating more challenging evaluation of nuanced understanding and reasoning in complex real-world tasks.Recent models approach or exceed 90% accuracy on existing visual grounding benchmarks.Source: §2 · pp. 2–3

Visual Grounding Datasets and Benchmarks

• Visual grounding benchmarks progressed from closed-set single objects and brief phrases toward open-vocabulary targets and complex descriptions.RefCOCO became largely saturated; later datasets addressed selected challenges, including compositional reasoning, crowded-scene discrimination, and rejection.Source: §2 · p. 3
• GroundingME is presented as the first benchmark to unify the critical dimensions for assessing advanced visual grounding in modern MLLMs.Earlier benchmarks addressed specific shortcomings but are described as still falling short of a sufficient challenge.Source: §2 · p. 3

3.1. Data Source

• The benchmark draws on SA-1B for all subtasks except Small, which uses HR-Bench; both contribute raw images without prior annotations.The source selection targets complex, dense scenes and ultra-high-resolution imagery.Source: §3.1 · p. 3

3.2. Data Construction

• The construction pipeline has three stages: bounding-box annotation, description generation, and manual selection and refinement.The overview presents this as a multi-stage process for constructing the benchmark.Source: §3.2 · p. 3

3.2.1. Bounding Box Annotation

• SA-1B boxes are generated with RAM++, GroundingDINO, and customized NMS, while HR-Bench boxes are manually annotated due to ultra-high resolution.The SA-1B pipeline identifies object categories, generates boxes, then favors classes with higher instance counts during NMS.Source: §3.2.1 · pp. 3–4

3.2.2. Description Generation

• Gemini-2.5-Flash generates preliminary descriptions, using full-image visual prompting for SA-1B and cropped regions for HR-Bench.SA-1B descriptions include visual attributes and spatial relations; HR-Bench descriptions focus on visual attributes.Source: §3.2.2 · p. 4

3.2.3. Manual Selection and Refinement

• Human annotators refine examples to address inaccurate boxes, hallucinated descriptions, and redundant or simplistic samples.They enforce uniqueness, subject clarity, task specificity, and factual accuracy; agreement on 50 samples averages Cohen’s kappa 0.69.Source: §3.2.3 · p. 4

3.3.1. Subtask Distribution

• The 1,005 samples are distributed across Discriminative (204), Spatial (300), Limited (300), and Rejection (201).The taxonomy comprises four L-1 categories and twelve L-2 subcategories.Source: §3.3.1 · p. 4, Figure 3 · p. 5

3.3.2. Statistics Analysis

• GroundingME includes 241 object classes, and its Intra-Class Count Quartile is (5, 7, 12), indicating many similar distractors.The quartile is derived from the automated construction pipeline excluding HR-Bench images and is a lower bound for the actual value.Source: §3.3.2 · p. 4, Table 2 · p. 5
• Description Length Quartile reaches (18, 40, 58) words, exceeding the reported averages for RefCOCO/+, RefCOCOg, and Ref-L4.Instance Area Ratio Quartile is (0.16%, 1.0%, 2.7%), indicating instances are small relative to images.Source: §3.3.2 · pp. 4–5

4.1. Evaluated Models

• The study evaluates 25 state-of-the-art MLLMs on GroundingME, with open-source models spanning 12 major publishers and a wide range of parameter sizes, plus commercial models chosen for explicit grounding ability.Source: §4.1 · p. 5

4.2. Evaluation Settings

• A unified prompt specifies the spatial reference viewpoint, allowed number of target objects, and required output format; evaluation uses greedy decoding and Accuracy@0.5.Accuracy@0.5 is the proportion of samples whose predicted and ground-truth boxes have Intersection over Union above 0.5.Source: §4.2 · p. 5

4.3.1. Main Results

• 45.1% accuracy is achieved by Qwen3-VL-235B-A22B, while most models score between 10% and 40%.Seed-1.6-Vision-250815 reaches 42.6%; accuracy rises with scale across Qwen3-VL-Dense (2B to 32B: 21.1% to 39.5%), Qwen3-VL-MoE (A3B to A22B: 35.7% to 45.1%), and Qwen2.5-VL (7B to 72B: 15.1% to 29.6%).Source: §4.3.1 · p. 5, Table 3 · p. 6

4.3.2. Subtask Results

• 75.0% versus 44.6% Average marks Qwen3-VL's 2B-to-32B Discriminative gain; models perform best on Discriminative and poorly on Rejection.Within Spatial, Relational outperforms Counting; Qwen3-VL (2B to 32B) improves Relationship from 11.3% to 61.3% and Average from 11.7% to 47.3%. State performance remains lower than the other three Discriminative subcategories.Source: §4.3.2 · p. 6

5.1. The Effectiveness of Thinking

• Thinking mode generally has a negative effect on tasks relying more on perception, yet shows notable performance improvements on tasks prioritizing reasoning; detailed subtask performance is provided in the Appendix.Without thinking, most models score 0% accuracy on Rejection. With thinking, all models demonstrate some rejection behavior, although performance remains low.Source: §5.1 · p. 7

5.2. Test-Time Scaling by Thinking Quality

• Test-time scaling selects among 16 generated responses through pairwise judge comparisons based on thinking quality.The study also compares judges given only final answers against judges with access to the thinking trajectories.Source: §5.2 · p. 7
• A 4.5% total gain across all subtasks follows when Qwen3-VL-A22B uses thinking trajectories for judge-based test-time scaling.Text-only judges also improve performance: DeepSeek-R1 by 2.9% and MiMo-7B-RL-0530 by 2.2%.Source: §5.2 · pp. 7–8, Table 4 · p. 7

5.3. Enhancing Rejection by Data Mixture

• The data-mixture experiments fine-tune Qwen3-VL-8B-Instruct on RefCOCOg-derived positive and modified-description negative samples across five negative-to-positive ratios.The resulting datasets contain 30,000 examples each, with ratios from 1:8 to 2:1; evaluation uses 11,490 curated negative validation samples.Source: §5.3 · p. 8
• Adding negative data enables rejection, and increasing its proportion improves rejection performance while affecting positive grounding to some extent.Across mixture ratios, macro average accuracy over positive and negative classes gains approximately 30%.Source: §5.3 · p. 8, Table 5 · p. 8
• 38.8% to a maximum of 33.0% on GroundingME w/o Rejection shows that rejection gains from data mixture do not transfer freely to harder out-of-domain grounding.Rejection performance on GroundingME rises as the negative-sample ratio increases, while performance excluding Rejection degrades relative to the pre-fine-tuned baseline.Source: §5.3 · p. 8, Figure 6 · p. 8

• GroundingME is a challenging benchmark that evaluates visual grounding through carefully curated samples across multiple dimensions.Source: §6 · p. 8
• The evaluation finds that current MLLMs struggle with scenarios that are more challenging than those in existing benchmarks.Source: §6 · p. 8
• Test-time scaling based on thinking quality and rejection-focused data-mixture training offer promising directions for improvement.Source: §6 · p. 8

• Table 6 presents subtask performance for different models with thinking mode enabled.Source: §A.1 · p. 12, Table 6 · p. 12
• Table 7 reports subtask performance for fine-tuned Qwen3-VL-8B-Instruct under different SFT data ratios.Source: §A.1 · p. 12, Table 7 · p. 12
• These tables supplement the analysis of thinking-mode gains and the out-of-domain results of data-mixture training.Source: §A.1 · p. 12

• Accuracy is reported at IoU thresholds 0.75 and 0.9, in addition to the main evaluation threshold.Source: §A.2 · p. 12
• The threshold-based evaluation covers all of GroundingME and the three L-1 categories other than Rejection.Source: §A.2 · p. 12
• mAcc is the mean accuracy over IoU thresholds from 0.5 to 0.95, sampled at intervals of 0.05.Source: §A.2 · p. 12

B.1. Cross-Dimensional Imbalance.

• Across the four L1 types, performance ranks Dis.>Spa.≈Lim.>Rej., with further imbalance across L2 subtasks.Sta. ranks lowest within Dis., while Cnt. ranks below Rel. within Spa.Source: §B.1 · p. 12, Figure 7 · p. 12

B.2. Subtask Ranking Difference.

• Seed-1.6-V. ranks #7 on Dis. Txt. but #1 on every Spatial task, illustrating substantial model-ranking divergence by dimension.The authors suggest these discrepancies may indicate training-data imbalances and directions for domain-specific enhancement.Source: §B.2 · p. 12, Figure 7 · p. 12

B.3. Model Family Behaviors.

• Across Qwen3-VL dense models from 2B to 32B, Rejection scores remain 0, while Discriminative and Spatial performance rise with scale and Limited shows no clear trend.Subtask decomposition indicates the Limited anomaly stems from Small.Source: §B.3 · p. 12, Figure 8 · p. 13

C.1. Prompt Templates

• The appendix lists a unified evaluation prompt and separate judge prompts for multimodal and text-only judges used in test-time scaling.The two judge prompts select among candidate responses in the test-time scaling analysis.Source: §C.1 · p. 13

C.2. Prompt Robustness

• Using five prompts, Qwen3-VL-8B and Llama-Nemotron score 31.36±2.02 and 10.07±0.44, respectively, with little variance from prior scores.98.61% of Qwen3-VL-8B responses follow the instructed format; rules were used to fix the remainder.Source: §C.2 · p. 13

D. Human Rejection Verification

• On a 100-instance human verification sample, annotators achieve 91% accuracy and 90.91% F1 for identifying Rejection examples.The sample includes 51 Rejection instances; annotators also achieve 88.24% precision and 93.75% recall.Source: D. Human Rejection Verification · p. 13

E. Commercial Model Notes

• For Gemini-2.5, the authors change the coordinate order to [y1, x1, y2, x2]; GPT-5, Claude-Sonnet-4.5, and Grok-4 are omitted because their outputs are unusable.The Gemini format change reflects greater receptiveness and measurable accuracy improvement; the other models showed substantial coordinate displacement and distortion, with no usable alternative format found.Source: E. Commercial Model Notes · p. 13

F. Tool Use Results

• 12.4% total accuracy is reported for Claude-Sonnet-4.5 with PyVision, including 7.3% on Limited Small.Its reported subtasks are Discriminative 19.1%, Spatial 13.3%, Limited 9.0%, and Rejection 9.5%.Source: F. Tool Use Results · p. 13
• Limited Small accuracy of 7.3% falls below expectations for multi-step cropping and magnification of 8K images, with subtle box-size and position offsets a significant contributing factor.The case study is presented in Table 13.Source: F. Tool Use Results · p. 13, Table 13 · p. 16

G. Examples for Each L-2 Subcategory

• The appendix defines all twelve L2 subcategories and presents one representative example for each.The examples mark the ground-truth object with a red bounding box.Source: G. Examples for Each L-2 Subcategory · p. 13