Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks

Qihua Dong; Kuo Yang; Lin Ju; Handong Zhao; Yitian Zhang; Yizhou Wang; Huimin Zeng; Jianglin Lu; Yun Fu

Source-linked AI summary

Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks

Qihua Dong, Kuo Yang, Lin Ju, Handong Zhao, Yitian Zhang, Yizhou Wang, Huimin Zeng, Jianglin Lu, Yun Fu

arXiv:2602.23898v1Feb 27, 2026cs.CVcs.AIcs.CL

TL;DR

Existing REC benchmarks can overstate multimodal models’ visual reasoning because they permit short expressions, few distractors, and shortcut cues. Ref-Adv introduces a harder benchmark requiring minimally sufficient expressions and evaluates contemporary MLLMs, revealing a significant performance drop relative to RefCOCO(+/g).

Problem
Classic REC benchmarks provide limited evidence of multistep textual and visual reasoning because expressions are short, distractors are few, and descriptors can enable shortcuts.
Method
Ref-Adv constructs a benchmark with minimally sufficient referring expressions, hard distractors, and ablations probing textual and visual reasoning.
Results
Contemporary MLLMs show a significant performance drop on Ref-Adv compared with near-saturated RefCOCO(+/g) scores, with GPT-4o achieving the best Ref-Adv performance under CoT with SoM.
Takeaways & Limitations
Ref-Adv exposes clear gaps in current MLLMs’ visual reasoning and grounding capabilities.
Takeaways & Limitations
The benchmark is intended for evaluating grounding and visual reasoning, not surveillance or biometric identification, and usage guidance discourages identity inference and sensitive-attribute prediction.

Abstract

from arXiv · show

Referring Expression Comprehension (REC) links language to region level visual perception. Standard benchmarks (RefCOCO, RefCOCO+, RefCOCOg) have progressed rapidly with multimodal LLMs but remain weak tests of visual reasoning and grounding: (i) many expressions are very short, leaving little reasoning demand; (ii) images often contain few distractors, making the target easy to find; and (iii) redundant descriptors enable shortcut solutions that bypass genuine text understanding and visual reasoning. We introduce Ref-Adv, a modern REC benchmark that suppresses shortcuts by pairing linguistically nontrivial expressions with only the information necessary to uniquely identify the target. The dataset contains referring expressions on real images, curated with hard distractors and annotated with reasoning facets including negation. We conduct comprehensive ablations (word order perturbations and descriptor deletion sufficiency) to show that solving Ref-Adv requires reasoning beyond simple cues, and we evaluate a broad suite of contemporary multimodal LLMs on Ref-Adv. Despite strong results on RefCOCO, RefCOCO+, and RefCOCOg, models drop markedly on Ref-Adv, revealing reliance on shortcuts and gaps in visual reasoning and grounding. We provide an in depth failure analysis and aim for Ref-Adv to guide future work on visual reasoning and grounding in MLLMs.

1 INTRODUCTION

REC grounds natural-language expressions to image regions and benchmarks fine-grained language–vision correspondence in MLLMs. Ref-Adv addresses classic benchmarks’ weak reasoning demands by preserving natural expressions while adding textual, visual, and grounding challenges.

REC and MLLM evaluation: REC grounds a natural-language expression to a specific image region and evaluates fine-grained correspondence between language and vision in MLLMs.The task has important applications in real-world systems and downstream tasks.
Ref-Adv motivation: Ref-Adv preserves the classic REC setting and natural expressions while avoiding short reasoning paths and imposing textual, visual, and grounding challenges on contemporary MLLMs.The benchmark is introduced to substantially increase reasoning challenge in line with modern LLM capabilities.
Limitations of classic benchmarks: Classic REC benchmarks often use very short expressions and few distractors, reducing required linguistic effort, descriptor verification, and candidate search.RefCOCO and RefCOCO+ expressions average around 3 words, while most RefCOCO(+/g) cases contain only 1 distractor.
Limitations of classic benchmarks: Longer expressions do not necessarily increase difficulty because redundant descriptors and few distractors enable grounding shortcuts based on matching only a subset of descriptors.Figure 2(a) illustrates that longer expressions can paradoxically yield higher accuracy under these conditions.

2 THE REF-ADV DATASET

Ref-Adv is curated from COCO and OpenImages with linguistically complex expressions, hard same-category distractors, and minimally sufficient descriptors to require visual reasoning. Its LLM-assisted, human-verified pipeline generates and validates diverse expressions, including negation-based alternatives and human-authored examples.

Data Sources and Representation: Ref-Adv samples COCO and OpenImages v7 images with panoptic instance annotations, converting bounding boxes to absolute [x1, y1, x2, y2] coordinates.The pipeline takes images, box annotations, and category names as input and outputs expressions paired with target instances.
Distractor Design: Images contain at least 3 same-category candidate instances, while hard distractors partially match but do not exactly satisfy the referring expression.These constraints reduce category-only grounding and shortcuts based on redundant descriptors.
Expression Generation: LLMs generate natural, diverse expressions from discriminative descriptors, using either target attributes or negated hard-distractor attributes.The two-stage design first elicits group-level and instance-level discriminators, then composes multiple candidate expressions.
Verification and Human Data: Human verification checks expression correctness, unambiguity, and hard-distractor presence after annotators attempt grounding on the untagged image.The dataset also includes a subset of human-authored expressions written with diverse and natural phrasing.
Dataset Statistics: Ref-Adv shows clear advantages over existing REC benchmarks in expression length, vocabulary size, distractor counts, and negation ratio.These statistics are reported across the validation and test sets of RefCOCO, RefCOCO+, RefCOCOg, and Ref-Adv.

3 EXPERIMENT

Experiments evaluate contemporary closed- and open-source MLLMs on Ref-Adv using model-specific prompts, multiple IoU thresholds, SoM-assisted grounding where needed, and CoT variants. Results show substantial difficulty with hard distractors, benefits from CoT, and marked gaps in visual reasoning and perception despite strong performance on standard REC benchmarks.

Evaluated Models: The evaluation covers contemporary closed- and open-source MLLMs, including Qwen2.5-VL, InternVL3, Gemini, CogVLM-Grounding, GLM-4.5V, GPT-4o, and Claude-3.5 Sonnet.The suite includes both model families and individual systems evaluated on Ref-Adv.
Evaluation Methods: GPT-4o and Claude-3.5 are evaluated with Set-of-Marks and Semantic-SAM because their grounding abilities are limited.SoM overlays numbered marks on candidate objects and uses Semantic-SAM for fine-grained localization.
Effect of Distractor Count: −2.7 Acc0.5 is the change for Qwen2.5-VL-72B+CoT in the ≥7 distractor group, compared with overall Acc0.5.Most models change modestly with 4–6 distractors but drop more with ≥7, indicating that many similar distractors remain challenging.
Effect of CoT: CoT generally improves Ref-Adv performance, reflecting the benchmark’s heavier reasoning demand, whereas CogVLM-Grounding does not support CoT.Each other model is evaluated with and without CoT; standard REC benchmarks may not require extensive reasoning.
Ref-Adv-s: 1,142 cases comprise the publicly released Ref-Adv-s subset with evaluation code, spanning Qwen2.5-VL, Qwen3-VL, and Qwen3.5 models from 2B to 397B parameters.On Ref-Adv-s, accuracy decreases as distractor count increases and thinking-mode variants substantially outperform instruct counterparts.
Main Results: GPT-4o achieves the best Ref-Adv performance under CoT with SoM, while other models’ accuracy drops markedly relative to RefCOCO(+/g).The result suggests gaps in visual reasoning and perception despite strong standard-benchmark performance.
Qualitative Analysis: Models often select hard distractors, or fail after incorrect visual perception or misunderstanding of the referring expression, even when explicit reasoning follows the intended chain.Qualitative examples for Qwen2.5-VL-72B and Gemini 2.5-Flash expose these failure modes with and without CoT.

4 LITERATURE REVIEW

The literature review situates Ref-Adv within foundational referring-expression benchmarks and the rapid rise of multimodal large language models. It highlights grounding as an increasingly important focus in visual-language research.

Referring Expression Benchmarks: Foundational referring-expression benchmarks include ReferItGame and the RefCOCO suite, which have advanced research in the field.The passage also places segmentation-based benchmarks within a broader category spanning diverse domains and applications.
Motivation: Rapid advances in multimodal large language models have increased the need for more challenging benchmarks.The passage describes MLLMs as now dominating the field.
Multimodal Large Language Models: Recent vision-language progress combines powerful language-model backbones, vision encoders, and alignment tuning for instruction following.A growing body of work studies visual understanding in large language models.
Multimodal Large Language Models: Grounding ability has emerged as an important research focus in multimodal large language models.The passage cites recent work examining grounding within broader efforts on visual understanding.

5 CONCLUSION

The paper introduces Ref-Adv, a REC benchmark designed to reduce visual shortcuts through minimally sufficient expressions and genuine multi-step reasoning. Ablations validate its focus on complex textual and visual grounding, while MLLM evaluations expose a significant drop relative to near-saturated RefCOCO(+/g) performance.

Ref-Adv targets reliance on visual shortcuts in existing REC datasets by requiring genuine multi-step reasoning.
A two-stage pipeline uses an LLM to compose minimally sufficient referring expressions for Ref-Adv.
The ablation studies confirm that Ref-Adv probes complex textual and visual grounding capabilities.
Contemporary MLLMs show a significant performance drop on Ref-Adv compared with their near-saturated scores on RefCOCO(+/g).The findings expose a critical overestimation of these models’ visual reasoning abilities.

ETHICS STATEMENT

The benchmark follows the ICLR Code of Ethics and uses human verification to support safe annotation, while defining its purpose as grounding and visual-reasoning evaluation rather than surveillance or biometric identification.

Ethical safeguards: The authors follow the ICLR Code of Ethics.They cite the ICLR Code of Ethics directly.
Ethical safeguards: Large language models draft candidate expressions, which three annotators verify for correctness and ambiguity or safety.The human verification step is described in Section 2.
Ethical safeguards: Annotators use only public images and may skip any example.This participation safeguard applies during annotation.
Intended use: The benchmark is intended to evaluate grounding and visual reasoning, not surveillance or biometric identification.The statement explicitly limits the benchmark’s intended use.

REPRODUCIBILITY STATEMENT

The paper supports reproducibility by documenting the complete data pipeline and committing to release the dataset artifacts, annotation schema, loading and evaluation scripts, and evaluation protocols.

Data pipeline: Section 2 details the data pipeline, including image sourcing, same-class distractor filtering, descriptor elicitation, expression composition, and three-annotator verification.A step-by-step pipeline diagram is provided in Figure 3.
Released artifacts: The authors will release image identifiers, final referring expressions, target regions, annotation JSON schema, and scripts for data loading and evaluation.The evaluation protocols and metrics include Acc0.5, Acc0.75, and Acc0.9.

V Team, Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale

This section lists prior work on referring-expression comprehension, language-based image segmentation, multimodal reasoning, visual grounding, and contemporary multimodal models. It also presents category-distribution comparisons across RefCOCO variants and Ref-Adv.

Related work: Prior work includes referring-expression comprehension benchmarks and language-based image segmentation in the wild.The cited works include a large-scale human-centric benchmark for referring expression comprehension and PhraseCut.
Related work: The references include chain-of-thought prompting and set-of-mark prompting for reasoning and visual grounding.The cited studies address reasoning in large language models and visual grounding in GPT-4V.
Related work: The section cites contemporary multimodal models including DeepSeek-VL2, Grok-4 fast, Qwen2.5, and InternVL3.These references cover mixture-of-experts vision-language models, a fast model release, a technical report, and advanced training and test-time recipes.
Category distribution: Figure 6 compares category-distribution ratio curves for RefCOCO, RefCOCO+, RefCOCOg, and Ref-Adv.The frequency ratio is plotted on a logarithmic scale after categories are sorted within each dataset.

B DATASET CATEGORY DISTRIBUTIONS

Ref-Adv exhibits a more long-tailed category-level frequency distribution than RefCOCO, RefCOCO+, and RefCOCOg. This comparison is visualized using sorted category frequency ratios on a logarithmic scale.

B DATASET CATEGORY DISTRIBUTIONS: Ref-Adv follows a more long-tailed category-level frequency distribution than RefCOCO, RefCOCO+, and RefCOCOg.Figure 6 compares sorted category frequency ratios across the four datasets on a logarithmic scale.

C PROMPT IN DATA COLLECTION

The data-collection pipeline uses two prompt stages to identify discriminators and compose minimally sufficient referring expressions, with structured JSON outputs enforcing format correctness.

Two-stage prompt pipeline: Query 1 elicits group-level and intra-pair discriminators, while Query 2 composes minimally sufficient referring expressions from them.Placeholders including {num objects} and {target class} are filled at runtime.
Output formatting: The LLMs produce structured JSON outputs to ensure the generated results follow the required format.

D LLM API COST FOR DATA COLLECTION

LLM-authored expression collection has a low 18.7% keep rate, making each retained expression cost substantially more than the initial generation cost. Using GPT-4o pricing, the effective cost per kept expression is $0.0372.

Cost calculation: 18.7% of LLM-authored expressions are kept, requiring approximately 5.35 generated expressions per retained expression.The passage computes 1/0.187 = 5.35.
Cost calculation: Each LLM-authored expression uses about 2300 input tokens and 120 output tokens.These token counts are used in the GPT-4o cost calculation.
Cost calculation: $0.00695 is the generation cost for one LLM-authored expression under GPT-4o pricing.The calculation uses $2.5 per 1M input tokens and $10 per 1M output tokens.
Cost calculation: $0.0372 is the effective cost per kept expression after accounting for the 18.7% keep rate.This equals 5.35 × $0.00695.