Recognizing Dialogue Ambiguity via Candidate Set Visual Grounding

• Interpreting visually grounded utterances often requires context beyond the utterance itself, making ambiguity handling relevant to real-world collaboration.An initial request such as “Can you pass me the drink?” may not identify a unique object from the utterance and scene alone.Source: §1 · p. 1
• CSVG evaluates all objects that remain plausible under the dialogue context, addressing cases where context cannot identify a unique target.Such candidate sets can support downstream actions such as asking clarifying questions.Source: §1 · p. 2
• CandVG treats each accumulated dialogue context as a separate evaluation sample to measure how candidate sets change across turns.The benchmark contains 1,249 dialogues and 4,833 evaluation samples.Source: §1 · p. 2
• Larger VLMs generally perform better on CVG, but increased model scale does not necessarily improve CSVG, where models often miss plausible candidates.The paper attributes incomplete predictions to candidate-generation difficulty and premature narrowing in early turns; CSVG-oriented instructions partially mitigate this for evaluated large-scale models.Source: §1 · p. 2

• Visual grounding localizes objects in an image that are referred to by a linguistic expression.Source: §2 · p. 2
• Interactive visual grounding uses dialogue context to identify which object an utterance refers to, an approach the paper calls Context-Aware Visual Grounding (CVG).Source: §2 · p. 2
• VLM-based grounding can incorporate textual context and flexibly interpret linguistic expressions, motivating the paper’s focus on VLM approaches for context-dependent grounding.Source: §2 · p. 2
• Unlike single-referent CVG, practical dialogue may leave multiple candidates plausible, and systems may identify candidates before asking clarification questions.CSVG evaluates whether systems identify the candidate set under available context; CandVG covers both CVG and CSVG.Source: §2 · pp. 2–3

3.1 Task Setting

• Given an image and dialogue context through turn i, the task predicts bounding boxes for objects still plausible as referents of the initial utterance.Source: §3.1 · p. 3, §3.2 · p. 3
• CVG applies when context identifies one object, and its goal is to predict that object’s bounding box.Source: §3.1 · p. 3
• CSVG applies when multiple objects remain plausible under context, and its goal is to predict their bounding boxes.Source: §3.1 · p. 3

3.2 Evaluation Metrics

• CVG uses Acc@0.5, counting a prediction correct when its bounding-box IoU with ground truth is at least 0.5.If multiple boxes are returned, only the first counts as the prediction.Source: §3.2 · p. 3
• CSVG matches predicted and ground-truth boxes one-to-one with the Hungarian algorithm, maximizing matches whose IoU is at least 0.5.Source: §3.2 · p. 3
• CSVG computes per-sample precision, recall, and F1 from matched pairs, unmatched predicted boxes, and unmatched ground-truth boxes, then averages the metrics over samples.Source: §3.2 · p. 3

4 CandVG

• CandVG extends an existing dialogue visual-grounding dataset with candidate sets for each context, yielding 1,249 dialogues and 4,833 evaluation samples.Annotators marked all plausible objects and mapped their points to the dataset’s bounding boxes.Source: §4 · p. 3

4.1 Data Collection

• Annotators updated candidate sets across turns by adding or removing points on plausible objects, then retained dialogues with complete point-to-box correspondence.The collection process began with 1,620 sampled dialogues, excluded 303 during screening, and retained 1,317 before correspondence filtering.Source: §4.1 · pp. 3–4

4.2 Statistics and Analysis

• CandVG contains 2,549 single-object CVG samples and 2,284 multi-object CSVG samples.Figure 2 presents candidate-set sizes per sample and unique-candidate counts per dialogue.Source: §4.2 · p. 4, Figure 2 · p. 4
• Two annotator pairs independently annotated 360 dialogues sampled across six candidate-count bins, and both pairs showed high agreement on CVG and CSVG metrics.Table 1 presents agreement for the two annotator pairs.Source: §4.2 · p. 4, Table 1 · p. 4

5.1 Settings

• The zero-shot evaluation covered generalist and embodied-reasoning VLMs, standard single- and multi-object grounding benchmarks, and CSVG with or without ground-truth candidates.The paired candidate conditions distinguish candidate generation from selection.Source: §5.1 · pp. 4–5

5.2 Main Results

• Embodied-reasoning post-training produced no consistent gains: both 2B models beat Qwen3 VL 2B on both tasks, but their 8B comparisons diverged by task.RynnBrain 8B did better on CVG but worse on CSVG; CosmosReason2 8B showed the reverse.Source: §5.2 · p. 5

5.3 Comparison with Standard Visual Grounding Benchmarks

• CandVG CVG accuracy was below RefCOCOg for all models and below F30k-Ent for most; its CSVG precision–recall gap exceeded gRefCOCO’s.The comparisons highlight the additional challenge of using dialogue context to infer plausible candidates.Source: §5.3 · pp. 5–6, Table 2 · p. 6, Figure 3 · p. 6

5.2 Main Results

• Across CandVG CSVG, larger open-weight models did not consistently improve F1, and recall remained below precision as model scale increased.The results indicate that models often identify precise subsets but miss plausible candidates.Source: §5.2 · p. 5, Table 2 · p. 6, Figure 3 · p. 6
• On CandVG CVG, accuracy rose with scale within each open-weight generalist family, with Qwen3.5 27B highest among evaluated models.Source: §5.2 · p. 5, Table 2 · p. 6

5.4 Effect of Ground-Truth Candidates

• Providing ground-truth candidates increased CSVG F1 for all tested models, with especially large gains for Intern3.5 variants, implicating candidate generation as a bottleneck.Source: Figure 4 · p. 6, §5.4 · p. 6

6.1 Tracking and Updating Candidate Sets

• Across three turns, Qwen3.5 27B narrowed candidates too early, Qwen3 VL 4B failed to narrow them sufficiently, and GPT-5.6 Luna moved from underprediction to overprediction.The analysis tracked the same 1,234 dialogues across CVG and CSVG cases.Source: §6.1 · p. 7, Figure 5 · p. 7
• Qualitative examples show Qwen3.5 27B committing to one object early but reaching the correct CVG target with more context, while GPT-5.6 Luna more closely followed updates overall.Qwen3 VL 4B retained multiple objects but failed to narrow the set in later turns.Source: §6.1 · p. 7, Figure 6 · p. 8

6.2 Effect of Task-Aware Instruction

• CSVG-specific instructions added a stepwise procedure and guidance to retain visually plausible referents unless text or dialogue context rules them out.Source: §6.2 · p. 7
• Both task-aware instructions improved recall for all 27B+ models, with larger gains from TAI-1; 4B recall gains were inconsistent, and effects did not consistently extend to precision.Source: §6.2 · pp. 7–8, Table 3 · p. 8

7 Conclusion

• CandVG pairs CSVG with CVG to evaluate candidate-set updates and ambiguity resolution across dialogue turns.CSVG identifies all candidates that remain plausible under the given context.Source: §7 · p. 8
• Model scale is associated with stronger visual grounding and higher CVG accuracy, but not consistently better CSVG F1 scores.Models often predict only a subset of candidates, producing a substantial precision–recall gap.Source: §7 · p. 8
• Candidate-generation difficulty and premature commitment are associated with the CSVG gap, while task-aware instructions improve recall for large-scale VLMs.Higher-recall models may still retain extra candidates as ambiguity resolves.Source: §7 · p. 8

Limitations

• CandVG inherits its source data’s image domains, dialogue styles, and ambiguity patterns; sudden candidate additions are underrepresented.Most dialogues progressively shrink candidate sets, often to one object.Source: Limitations · p. 9
• Static images and bounding boxes omit environmental change, viewpoint-dependent references, and non-verbal cues; one box may poorly represent object parts or regions.The authors identify dynamic, embodied, and potentially real-world environments as future directions.Source: Limitations · p. 9
• Open-weight evaluation reaches at most 38B parameters and proprietary evaluation omits major families, limiting generalization of observed scaling trends.The authors call for broader evaluation across architectures, sizes, and training objectives.Source: Limitations · p. 9
• Improvement experiments cover only inference-time task-aware instructions, not more sophisticated prompting or decoding strategies.Instructions do not yield consistent gains for compact VLMs, which the paper notes are preferable for deployment because of lower computational and latency requirements.Source: Limitations · p. 9

Ethical Consideration

• CandVG builds on an existing CVG dataset, uses Open Images images provided under Creative Commons Attribution licenses, and its newly collected annotations will be released under CC BY 4.0.Open Images annotations are released under CC BY 4.0.Source: Ethical Consideration · p. 9

A Data Collection Details

• CandVG uses InViG 21K’s natural human–human dialogues and image bounding boxes to guide selection, annotation, and point-to-box mapping.The final dataset retains dialogues with a complete one-to-one correspondence between annotated points and unique boxes at every turn.Source: A Data Collection Details · pp. 12–13
• Selection filters excluded dialogues lacking initial ambiguity or containing absent, non-specific, invalid, or uniquely identifiable referents.The authors also excluded group boxes of four or more objects; annotators could place points beyond the initially supplied boxes and add or remove them as context updated.Source: A Data Collection Details · pp. 12–13

B Additional Statistics and Analysis

• Many CandVG samples converge to a single candidate by the third turn as dialogue context progressively resolves initial ambiguity.In 21 dialogues (1.68%), multiple candidates remain plausible at the final turn, and those samples are included in CSVG evaluation.Source: B Additional Statistics and Analysis · p. 13
• Annotation heatmaps are concentrated near image centers but span the image; later-turn densities are lower as candidate sets shrink.The heatmaps aggregate normalized point coordinates into smoothed two-dimensional histograms.Source: B Additional Statistics and Analysis · p. 13

C Implementation Details

• Open-weight models used greedy decoding with a 512-token maximum and normalized box coordinates, while GPT-5.6 used a 2,048-token maximum and image-relative coordinates.GPT-5.6 reasoning effort was medium.Source: C Implementation Details · p. 13

D.1 Results on CVG

• Qwen3 VL 4B predicts multiple bounding boxes for 74.2% of CVG samples, with lower precision than recall.This pattern is consistent with its tendency to predict extra candidates.Source: §D.1 · p. 14, Table 7 · p. 16
• CVG accuracy substantially improves from Turn 1 to Necessary across all models; Full is generally comparable to Necessary and slightly higher for most.The comparison tests first-utterance-only, minimum sufficient, and full dialogue context.Source: §D.1 · pp. 14–15, Table 8 · p. 16

D.2 Results on CSVG

• For many open-weight models, CSVG F1 tends to decrease as candidate-set size increases, while GPT-5.6 remains stable.The authors attribute the open-weight trend likely to lower recall.Source: §D.2 · p. 15, Table 10 · p. 18

E Additional Transition Analysis

• Across models, the Not Predicted rate exceeds Not Removed, indicating many candidates removed from the ground-truth set were not predicted at the earlier turn.This pattern is consistent with candidate-generation difficulty and premature commitment while utterances remain ambiguous.Source: E Additional Transition Analysis · p. 17, Table 11 · p. 18