PaperlayerView exact revision on arXiv

Source-linked AI summary

When Seeing Is Not Enough: Benchmarking Interactive Visual Grounding in LVLMs

Zhengxiang Wang, Owen Rambow

arXiv:2608.23978v1cs.AIcs.CV

TL;DR

Standard grounding assumes an informative description, leaving interactive reference underrepresented despite ambiguity and clarification in real communication. The paper introduces a controlled LVLM evaluation framework varying upfront target information and dialogue requirements. Across contexts and protocols, current LVLMs trail task-level human references, benefit from some clarification, but struggle with proactive question-driven grounding and calibration.

  • Problem

    One-shot grounding assumes an informative referring expression and does not test recognizing incomplete descriptions, seeking missing information, or updating decisions.

  • Method

    The paper evaluates LVLM Matchers across four human-grounded visual contexts and four protocols that vary supplied versus dialogue-acquired target information.

  • Results

    Current LVLMs perform significantly below task-level human baselines; interaction can refine or repair descriptions, but proactive question-driven grounding remains difficult.

  • Takeaways & Limitations

    Interactive visual grounding remains a challenge requiring visual matching, information seeking, and synthesis beyond static matching.

  • Takeaways & Limitations

    The simulated Director enables scale and control but does not establish equivalence to human Directors, and outcomes can vary with the Director.

Abstract

from arXiv · show

Visual grounding is typically evaluated as a one-shot mapping from an informative referring expression to a visual target. This formulation misses a central property of real-world reference: target information is often incomplete, ambiguous, and established through interaction. We introduce a controlled evaluation framework for interactive visual grounding in large vision-language models (LVLMs), varying how much target information is provided upfront and how much must be acquired through dialogue. Across four human-grounded visual contexts and four interaction protocols, current LVLMs perform significantly below task-level human baselines. Interaction can help when follow-up questions refine or repair an initial target description. Performance is lowest when no initial description is provided and target information must be acquired through questions, indicating that proactive question-driven grounding remains difficult. LVLMs are also poorly calibrated, often reporting confidence that exceeds their empirical accuracy. Follow-up studies confirm these patterns across varied description sources (human versus AI), reasoning efforts, repeated interactions, description providers, and visual contexts. Overall, interactive visual grounding remains an important challenge, requiring visual matching, information seeking and synthesis.

1 Introduction

Traditional visual grounding assumes an informative description and evaluates one-shot visual matching, whereas real-world reference often requires clarification and shared information-building. The paper introduces controlled interactive evaluation to test these additional capabilities in LVLMs.

  • Traditional benchmarks map an already informative referring expression to a visual target in one shot.
  • Real-world reference is often underspecified or ambiguous and is established collaboratively through clarification and feedback.
  • Interactive visual grounding requires visual matching plus information seeking and synthesis across dialogue turns.
  • Prior work does not systematically separate failures to match visual targets from failures to acquire information.
  • The framework varies upfront target information and dialogue requirements, and evaluates 38,848 target selections from eight LVLMs.

2 Related Work

Related work covers static grounding, interactive reference games, human–AI communication, and multi-turn evaluation. These studies motivate interactive assessment but leave systematic separation of matching and information-seeking abilities unresolved.

  • Static Visual Grounding: Static visual grounding primarily measures selecting a corresponding region or object from an image and referring expression.
  • Interactive Visual Grounding: Interactive settings include yes/no object-identification games, clarification-based robot grounding, and open-ended multi-turn disambiguation.
  • LVLMs in Reference Games: Reference-game studies examine repeated human dialogues, LVLM self-play, and human–AI interaction.
  • Multi-Turn Evaluation: Multi-turn evaluation tests whether language and vision-language models track dialogue history and evolving user intent across turns.

3 Interactive Visual Grounding

The task models a Matcher and Director who exchange utterances under partial observability until the Matcher selects targets, optionally revises decisions, and varies how much information is initially available. Accuracy, question rate, and calibration measure complementary behaviors.

  • Task Formalization: The Director sees an ordered target set, while the Matcher sees a separately indexed candidate set containing targets and possible distractors.
  • Task Formalization: For each target, players exchange utterances until the Matcher selects a candidate, with clarification and later revision allowed.
  • Interaction Protocols: The four protocols progress from static informative-description matching to optional clarification, underspecification recovery, and fully question-driven grounding.
  • Interaction Protocols: FULL permits follow-up questions to verify, refine, or repair an initially informative description, while PARTIAL begins with an underspecified attribute.
  • Evaluation Metrics: Accuracy measures correct matches, question rate measures average questions per item, and calibration compares stated confidence with empirical accuracy.

4 Experimental Setup

The experiments use four human-grounded visual contexts, filtered and transformed into descriptions for four interaction protocols, with LVLMs evaluated against controlled Director responses. The setup includes extensive simulations and follow-up analyses.

  • Datasets: The evaluation spans four human-grounded object-level contexts covering varied visual ambiguity, nameability, and referential difficulty.
  • Datasets: Initial-round dialogues were filtered using human matching thresholds and participant performance across repeated rounds to control description quality and stability.
  • Description Construction: GPT-5.4 extracted 536 full referring expressions, while least-informative attributes were extracted to construct underspecified PARTIAL descriptions.
  • Director Simulation: A fixed GPT-5.4 simulated Director provides comparable information access across Matchers and protocols, with sensitivity tested using an alternative Director.
  • Simulation Scale: Eight LVLM Matchers produce 17,152 main-experiment selections and 38,848 selections including follow-up studies.
  • Simulation Procedure: The Matcher receives indexed candidate grids, follows protocol-specific turn-taking, may ask up to the 10-turn budget, and can revise selections.

5 Results

Interactive visual grounding remains difficult for LVLMs: models generally trail human matchers, while interaction helps mainly when refining available descriptions rather than constructing targets from scratch.

  • 5.1 Interactive Visual Grounding Poses a Challenging Task: LVLM matchers generally underperform task-level human baselines across four image sets, with the largest gaps on the Tangram Image Set.These gaps are often substantial and statistically significant; Gemini-3.1-Pro reaches parity only on Basket Image Set 2 under PARTIAL.
  • 5.1 Interactive Visual Grounding Poses a Challenging Task: Success requires useful questions and effective integration of answers, not merely asking more questions.GPT-5.5 outperforms Gemini-3.1-Pro under NONE despite asking fewer questions, while GPT-5.4-Mini’s advantage varies by protocol.
  • 5.2 Stronger Models Perform Better, but Question-Only Grounding Remains Hard: Larger models generally outperform smaller models in the same family, and proprietary models tend to outperform open-weight models.Gemini-3.1-Pro is strongest overall under STATIC, FULL, and PARTIAL; GPT-5.5 is particularly competitive under NONE.
  • 5.1 Interactive Visual Grounding Poses a Challenging Task: FULL improves over STATIC for seven of eight models, but all models perform below STATIC under NONE despite asking the most questions.Optional follow-up questions can refine or repair initial descriptions, whereas proactive question-driven grounding remains difficult.
  • 5.3 LVLMs Are Not Yet Calibrated for Interactive Visual Grounding: Models ask more questions as initial information decreases, following NONE > PARTIAL > FULL > STATIC, but reported confidence often exceeds empirical accuracy.Overconfidence is especially pronounced under PARTIAL and NONE, where judging information sufficiency is less reliable.
  • 5.4 Summary: Only 49 self-repairs were observed, with 26 correcting wrong selections, 5 making correct selections wrong, and 18 replacing one wrong answer with another.Repair outcomes are strongly model-dependent: Gemini-3.1-Pro corrected 13 of 17 revisions, while GPT-5.4 corrected none of six.

6 Follow-Up Studies

Follow-up studies reproduce the main protocol pattern across description sources, reasoning settings, repeated interactions, Director models, and face-matching contexts. Clarification preserves performance when descriptions are informative, but performance declines as initial information is removed.

  • 6 Follow-Up Studies: AI-generated descriptions generally yield higher LVLM accuracy than human-generated descriptions, while STATIC and FULL remain strongest and NONE remains hardest.The protocol-level ordering remains comparable despite the description-source change.
  • 6 Follow-Up Studies: Reasoning substantially improves accuracy in FULL but provides little benefit in STATIC, with gains largely plateauing from medium to high effort.Reasoning also modestly improves calibration, with a similar medium-to-high-effort plateau.
  • 6 Follow-Up Studies: Models can improve over repeated rounds under FULL, but improvements are inconsistent across datasets and the gap to human matchers remains.The results suggest that models can handle increasingly efficient referring expressions emerging through entrainment.
  • 6 Follow-Up Studies: Replacing the GPT-5.4 Director reduces mean Matcher accuracy by 1.1 points in FULL and 3.5 points in PARTIAL, while rankings remain highly correlated.The larger PARTIAL decrease is consistent with the Director carrying more responsibility when initial descriptions are underspecified; rankings correlate at ρ = .929 for FULL and .976 for PARTIAL.
  • 6 Follow-Up Studies: In face matching, FULL remains within 0.9 points of STATIC at 47.8% versus 48.7%, while PARTIAL and NONE fall 6.8 and 18.0 points below STATIC.This extends the graded protocol pattern to a distinct fine-grained visual domain.

7 Conclusion

The paper introduces interactive visual grounding as an evaluation challenge beyond one-shot matching. Across varied contexts and protocols, LVLMs trail human references, struggle with proactive grounding, and often overstate confidence.

  • 7 Conclusion: The framework evaluates interactive visual grounding beyond one-shot matching across four human-grounded visual contexts.Follow-up studies vary description sources, reasoning effort, repeated interaction, simulated Directors, and visual contexts.
  • 7 Conclusion: Current LVLMs perform significantly below task-level human references.This pattern holds across the four human-grounded visual contexts examined in the main evaluation.
  • 7 Conclusion: Interaction helps when follow-up questions refine or repair an initial description, but proactive question-driven grounding remains difficult.The challenge is greatest when target information must be acquired through interaction rather than supplied initially.
  • 7 Conclusion: LVLMs are poorly calibrated, often reporting confidence that exceeds their empirical accuracy.The paper identifies overconfidence as a recurring property of interactive grounding behavior.
  • 7 Conclusion: Interactive visual grounding requires visual matching together with information seeking and synthesis.The conclusion frames these capabilities as central to future embodied human–AI interaction.

Limitations

The study’s conclusions are bounded by its simulated Directors, protocol-level attribution, controlled visual contexts, constrained NONE interaction, and use of existing visual materials.

  • Simulated Directors and Director Dependence: Simulated Directors provide scale and experimental control but do not establish equivalence to human Directors.Individual model–protocol outcomes can still vary with the Director.
  • Attribution: The protocols increase information-seeking demands but do not uniquely isolate question selection, answer integration, dialogue memory, or visual matching.Attributing failures to one precise subcomponent is left to future studies.
  • Visual Context Coverage: The visual testbed omits cluttered scenes, spatial or relational grounding, and open-world visual search.Conclusions therefore concern proactive interactive grounding in controlled, fine-grained matching contexts.
  • Constrained Interaction under NONE: Under NONE, the Director gives only yes/no answers, so open-ended answering strategies remain untested.The paper recommends studying open-ended Director answers separately.
  • Data Use and Privacy: The work uses existing datasets and public or previously released visual materials rather than collecting new human-subject data.The face study uses public images for within-set matching and does not infer identity or sensitive attributes.

B Expected Calibration Error

Expected Calibration Error measures the alignment between stated confidence and empirical accuracy by aggregating confidence–accuracy gaps across confidence bins. The paper reports calibration plots primarily because ECE is strongly correlated with accuracy in this setting.

  • B Expected Calibration Error: Expected Calibration Error summarizes alignment between the Matcher’s stated confidence and empirical accuracy.Each final selection receives a confidence score from 0 to 100, and predictions are grouped into ten evenly spaced bins.
  • B Expected Calibration Error: For each confidence bin, the method computes average empirical accuracy and average stated confidence.The bin-level quantities are then combined into a weighted average absolute gap.
  • B Expected Calibration Error: Lower ECE indicates better calibration between confidence and correctness.N denotes total predictions, |Bℓ| the number in bin Bℓ, acc(Bℓ) bin accuracy, and conf(Bℓ) bin confidence.
  • B Expected Calibration Error: ECE is strongly negatively correlated with accuracy in this setting, with Spearman’s ρ = −0.96.Because this makes ECE less informative as a separate measure, the main text emphasizes calibration plots.

C Extracted Partial Description Validation

The validation procedure constructs source-grounded underspecified descriptions by extracting minimally informative attributes and checking their lexical support and coverage. The extracted attributes are strongly supported by the original descriptions.

  • C Extracted Partial Description Validation: The validation computes attribute support using the maximum of exact substring matching and token overlap.Text is lowercased, punctuation is ignored, and only alphanumeric tokens are retained.
  • C Extracted Partial Description Validation: An attribute is supported when its support score s(ai, d) ≥ 0.45.The attribute support rate is the fraction of generated attributes exceeding this threshold.
  • C Extracted Partial Description Validation: Description coverage measures how much of the original full description is recovered by the generated attributes.The least-attribute support score separately verifies that each intentionally underspecified initial expression remains grounded in the full description.
  • C Extracted Partial Description Validation: Mean attribute support rate is 0.998, mean attribute support score is 0.980, and mean description coverage is 0.895.These results indicate that the decomposition preserves most descriptive content while keeping attributes grounded in source descriptions.
  • C Extracted Partial Description Validation: The extracted partial descriptions are source-grounded rather than hallucinated or detached from the original human descriptions.The validation therefore supports using them as intentionally underspecified PARTIAL inputs.

D.2 Self-Repair Analysis

Self-repair is rare and limited: revisions occur only under FULL and PARTIAL, with mixed effects on correctness and confidence. Follow-up analyses also show persistent difficulty, overconfidence, and broadly stable protocol patterns across conditions.

  • Self-Repair Analysis: Self-repairs account for no more than 2% of 536 matches in any protocol and occur only under FULL and PARTIAL.No self-repairs were observed under NONE.
  • Self-Repair Analysis: When models revise selections, repairs can correct errors, introduce errors, or replace one wrong answer with another.About half of GPT-5.5 repairs replace one wrong answer with another, and confidence changes are mixed.
  • Follow-Up Patterns: STATIC and FULL remain strongest, PARTIAL is more variable, and NONE remains difficult across follow-up conditions.These protocol-level patterns persist even when descriptions are AI-generated.
  • Reasoning Effort: Higher reasoning effort modestly increases question asking for GPT-5.4 under FULL, while GPT-5.4-Mini asks very few questions across settings.The results suggest reasoning may support information seeking without simply increasing question volume.
  • Calibration: Higher reasoning effort slightly improves calibration, but gains plateau from medium to high and most bins remain below the perfect-calibration diagonal.Both GPT-5.4 and GPT-5.4-Mini therefore remain overconfident.
  • Repeated Interaction: Repeated exposure can improve accuracy without consistently improving uncertainty estimates.Many confidence bins remain below the diagonal across rounds, indicating persistent overconfidence.

E.5 Face-Domain Robustness

A face-image robustness experiment extends the main protocol pattern to controlled within-category matching. STATIC and FULL are comparable, PARTIAL is lower, and NONE is most difficult, but the setting does not establish broader visual generality.

  • Experimental Setup: The robustness experiment evaluates five representative Matchers on six FACES image sets, each containing 12 images of one individual with varied facial expressions.Initial descriptions are generated with GPT-5.4, and all four interaction protocols are evaluated.
  • Results: 48.7% and 47.8% are the aggregate accuracies for STATIC and FULL, respectively, compared with 41.9% for PARTIAL and 30.7% for NONE.The experiment has no protocol-matched human baseline.
  • Scope: The face study extends the protocol-level observation beyond dogs, baskets, and tangrams but remains a controlled within-category matching context.It does not establish generality to cluttered scenes, spatial relations, or open-world visual search.
Loading 2608.23978v1…