PaperlayerView exact revision on arXiv

Source-linked AI summary

What Users Leave Unsaid: Under-Specified Queries Limit Vision-Language Models

Dasol Choi, Guijin Son, Hanwool Lee, Minhyuk Kim, Hyunwoo Ko, Teabin Lim, Ahn Eungyeol, Jungwhan Kim, Seunghyeok Hong, Youngsook Song

arXiv:2601.06165v2cs.CVcs.AI

TL;DR

Existing VLM benchmarks underrepresent the informal, underspecified visual questions users ask in practice. The paper introduces HAERAE-Vision and paired explicit rewrites to measure this issue, finding that explicitation improves performance substantially and exposes a gap between benchmark evaluation and real-world deployment.

  • Problem

    Current VLM benchmarks predominantly use clean, explicit prompts despite real users often omitting context and relying on images to convey it.

  • Method

    The paper constructs HAERAE-Vision from 653 authentic Korean community questions and pairs each with an explicit rewrite for controlled evaluation.

  • Results

    Query explicitation yields up to 22 point improvements across models, with smaller models benefiting most, while benchmark difficulty substantially reflects query under-specification.

  • Takeaways & Limitations

    The findings reveal a critical gap between benchmark evaluation and real-world deployment because natural user-query under-specification materially affects VLM difficulty.

Abstract

from arXiv · show

Current vision-language benchmarks predominantly feature well-structured questions with clear, explicit prompts. However, real user queries are often informal and underspecified. Users naturally leave much unsaid, relying on images to convey context. We introduce HAERAE-Vision, a benchmark of 653 real-world visual questions from Korean online communities (0.76% survival from 86K candidates), each paired with an explicit rewrite, yielding 1,306 query variants in total. Evaluating 39 VLMs, we find that even state-of-the-art models (GPT-5, Gemini 2.5 Pro) achieve under 50% on the original queries. Crucially, query explicitation alone yields 8 to 22 point improvements, with smaller models benefiting most. We further show that even with web search, under-specified queries underperform explicit queries without search, revealing that current retrieval cannot compensate for what users leave unsaid. Our findings demonstrate that a substantial portion of VLM difficulty stem from natural query under-specification instead of model capability, highlighting a critical gap between benchmark evaluation and real-world deployment.

1 Introduction

Real users often ask informal, underspecified visual questions that rely on images to supply omitted context, while standard VLM benchmarks largely use explicit prompts. HAERAE-Vision pairs authentic under-specified questions with explicit rewrites to measure this gap and shows that explicitation substantially improves performance.

  • Real users often omit context and rely on images, whereas current VLM benchmarks predominantly use clean, explicit prompts.This mismatch makes under-specification a central challenge for evaluating visual question answering in realistic interactions.
  • Up to 22 point improvements follow from query explicitation, with smaller models benefiting most dramatically.The result indicates that ambiguity substantially affects measured VLM performance.
  • Even state-of-the-art models score under 50% on original queries, while explicitation raises their performance above 55%.GPT-5 improves from 48.0% to 57.6%, and Gemini 2.5 Pro from 48.5% to 56.7%.
  • The findings attribute a substantial portion of VLM difficulty to natural query under-specification and identify a gap between benchmark evaluation and deployment.They challenge the assumption that benchmark difficulty solely reflects model capability limitations.
  • HAERAE-Vision contains 653 user-generated visual questions filtered from 86K candidates, spanning 13 domains.The benchmark retains authentic, informal query characteristics after rigorous filtering.
  • A parallel dataset of paired explicit rewrites enables controlled measurement of how omitted information affects VLM performance.The rewrites state missing information while preserving the original evaluation target.

2 HAERAE-Vision Benchmark

HAERAE-Vision is built from noisy, authentic Korean community questions through a multi-stage filtering and validation pipeline. Its paired explicit version preserves intent while making image-dependent context explicit for controlled evaluation.

  • 2 HAERAE-Vision Benchmark: HAERAE-Vision is designed to capture the under-specified, informal nature of real-world visual questions.The benchmark preserves natural user-query characteristics while converting noisy community data into evaluation problems.
  • Dataset Construction Pipeline: 86,052 question-image pairs from nine Korean platforms are reduced to 653 high-quality problems, a 0.76% survival rate.The source platforms cover general Q&A, gaming, science, and coding forums.
  • Dataset Construction Pipeline: The six-stage pipeline collects valuable items, filters inappropriate or trivial questions, verifies image dependence, generates checklists, and performs human validation.Automated screening is supplemented by model-based difficulty and image-dependence checks plus Korean annotator review.
  • Dataset Statistics: The final benchmark averages 3.3 checklist items and 1.3 images per question across 13 categories.Natural Objects and Gaming are the most represented categories, while platform survival rates range from 0.2% to 14.4%.
  • HAERAE-Vision-Explicit: Each question is rewritten to state missing information explicitly while preserving the original intent and scope.Rewriting replaces vague references, incorporates relevant visual information, and undergoes human validation.
  • HAERAE-Vision-Explicit: Representative examples span Daily Life, Gaming, and IT/Software, contrasting image-dependent vague references with clarified requests.The examples illustrate how background information is added to make the user request explicit.

3 Evaluation Framework

The evaluation framework uses detailed, instance-specific checklists to score partial understanding, reduce subjectivity, diagnose weaknesses, and support scalable automated evaluation.

  • Detailed checklists decompose complex answers into explicit criteria for fine-grained, reproducible evaluation.The approach supports partial-understanding assessment and reduces subjectivity through explicit criteria.
  • Each checklist item receives a score of 1.0, 0.5, or 0.0 based on explicit evidence in the model response.The judge also provides an extracted evidence line and a short justification for each score.
  • Overall performance averages instance-level checklist means, ensuring comparability across problems with different checklist lengths.

4 Experimental Setup

The experiments evaluate a broad set of proprietary and open-source VLMs on culturally specific visual questions using repeated runs and standardized sampling settings.

  • The evaluation covers proprietary and open-source VLM families across a broad range of model scales.Proprietary systems include GPT-5 and Gemini variants, while open-source models include Gemma, Qwen, Skywork, and InternVL families.
  • 45 VLMs are evaluated across the experimental model pool.
  • HAERAE-Vision includes culturally grounded examples such as Seoul subway interfaces, Korean calligraphy, drama scenes, TV settings, and historical registries.These items require knowledge that is rarely represented in English-centric datasets.
  • Each model is evaluated three times and averaged using standardized sampling parameters.The setup uses temperature, top_p, and max_tokens settings across models, with GPT-5 using a provider-constrained temperature.

5 Results

Original under-specified queries remain difficult for VLMs, while explicit rewrites substantially improve performance; web search helps less than clarification and cannot compensate for missing intent.

  • Baseline performance: 48.5% is the best reported original-query accuracy, achieved by Gemini 2.5 Pro, while GPT-5 reaches 48.0%.Neither search-augmented nor reasoning-optimized paradigms fully solve authentic, culturally grounded multimodal queries.
  • Query explicitation: 7.8 to 21.7 points are gained from query explicitation across six models, with GPT-5-Nano improving from 21.2 to 43.0.GPT-5 and Gemini 2.5 Pro gain 9.6 and 8.1 points, respectively, while GPT-5 reaches 57.6% after explicitation.
  • Web search: +7.57 and +5.87 points are gained from web search on original queries for GPT-5 and GPT-5-Mini, respectively.These gains are smaller than explicitation alone, which yields +9.56 and +7.83 points for the same models.
  • Web search: 55.58 remains below 57.57 for GPT-5, and 51.08 remains below 53.04 for GPT-5-Mini, comparing original-plus-search with explicit queries without search.The comparison indicates retrieval cannot compensate for under-specified queries.
  • Cross-lingual pilot: +3.2 to +6.6 points are gained in the English pilot, smaller than the Korean gains of +7.8 to +21.7.GPT-5-Nano scores 44.4% on English original queries versus 21.2% on Korean original queries.

6 Additional Analysis on Explicitation

Additional analyses show that explicitation removes surface ambiguity, exposes previously masked grounding failures, and shifts remaining errors toward cultural knowledge gaps.

  • What Explicitation Fixes: Lack-of-explicitness failures decline from 84.3% to 69.7%, a reduction of 14.6 percentage points after explicitation.
  • What Explicitation Fixes: GPT-5-Nano has 83 fewer error cases and a 12.7-point perfect-rate increase after explicitation, compared with GPT-5-Mini’s 40 fewer cases and 6.1-point increase.The larger reduction for the smaller model supports disproportionate sensitivity to under-specification.
  • Category-level effects: Mathematics, Science, Coding, and Shopping show the largest explicitation gains, whereas Natural Objects and Entertainment remain at a 0% all-models-pass rate.Remaining failures in the latter categories shift toward visual-text grounding and cultural knowledge gaps.
  • Error shifts: Visual-text grounding errors rise from 5.2% to 16.6% after explicitation, but 87% were already errors under original queries.The increase reflects previously masked grounding failures being reclassified or exposed.
  • Root causes: +6.4 percentage points in cultural-knowledge attribution accompany explicitation, making Korea-specific knowledge the dominant remaining challenge.The analysis reports cultural knowledge as more prominent once query ambiguity is resolved.

7 Reliability of LLM-as-a-Judge

The evaluation uses multiple LLM judges and checks their agreement with both one another and human annotators. Correlations are consistently high, supporting a stable and human-aligned judging signal.

  • Pearson correlations among four LLM judges range from 0.863 to 0.903, while Spearman correlations range from 0.866 to 0.901.
  • Human evaluations of 250 GPT-5-Mini judgments show high agreement, with Pearson r = 0.820 and Spearman ρ = 0.810 (p < 0.001).
  • Most low-agreement cases arise from superficial keyword matching or excessive leniency.

8 Related Work

Prior work evaluates ambiguity handling, clarification, and multimodal reasoning, while Korean research has developed localized text and vision-language benchmarks. HAERAE-Vision extends this landscape toward under-specified visual queries.

  • Existing VLM evaluations increasingly separate recognition, OCR, and knowledge from higher-level reasoning and instruction following.
  • Korean multimodal evaluation includes KRETA, KViscuit, KOFFVQA, KMMB, KSEED, and KDTCBench alongside Korean VLM development.
  • Ambiguity-focused resources measure answering choices, clarification, uncertainty reduction, or minimal question asking in conversational settings.

9 Conclusion

HAERAE-Vision benchmarks authentic Korean visual questions paired with explicit rewrites and finds a substantial performance gap caused by query underspecification. Retrieval does not close this gap, while remaining failures also involve cultural knowledge.

  • 653 authentic Korean questions are paired with explicit rewrites in HAERAE-Vision.
  • 8–22 points separate VLM performance on underspecified and explicit queries, and search-augmented underspecified queries still underperform explicit queries without search.
  • Many remaining failures reflect missing cultural knowledge rather than surface-level ambiguity.
  • An English pilot supports cross-lingual generalization of the explicitation effect, but its deltas are smaller because of fewer cultural knowledge barriers.
  • The benchmark prioritizes quality over quantity and is designed to study behavior under different prompt information densities rather than comprehensively evaluate Korean knowledge.

Appendices

The appendices document data collection across Korean platforms and provide platform-level descriptions and filtering statistics. The source platforms were selected to represent diverse users, expertise levels, and domains.

  • Data were collected from nine Korean online platforms representing diverse user communities and domain expertise.
  • Table 9 lists the Korean online platforms used for data collection.
  • The platform selection targets diversity in user demographics, expertise levels, and domain-specific knowledge.
  • Table 10 reports data collection and filtering statistics across platforms for Stages 1–6, with Inflearn, Codeit, and Okky combined as coding platforms.

A.3 English Pilot Study Details

The English pilot applies the Korean benchmark’s six-stage construction and explicitation protocol to publicly available Stack Exchange questions. Across four models, explicitation improves performance unevenly across domains, with Coding gaining most.

  • Data Source: The pilot collects 2,954 image-containing Q&A pairs from 12 Stack Exchange communities under CC BY-SA 4.0.
  • Filtering and Explicitation: The same six-stage pipeline filters candidates and applies protocol-consistent explicitation with human verification, selecting 100 stratified samples.Image-dependency verification removes over 60% of candidates whose images are supplementary rather than essential context.
  • Dataset Statistics: The final pilot contains 100 samples, 168 images, and an average of 3.4 checklist items per question.
  • Per-Domain Results: +14.4 is the largest average explicitation gain in Coding, while Natural Objects shows smaller improvements.These per-domain effects are averaged across all four evaluated models.
  • Evaluation Criteria: The benchmark’s checklist criteria are generated from satisfactory reference answers and require models to satisfy every listed criterion for correctness.GPT-4-mini generates structured criteria, which are intended as strict, human-aligned evaluation standards.
  • Judge Reliability: Judge feedback identifies superficial keyword matching, leniency toward incomplete responses, and difficulty separating implicit intent from explicit requirement satisfaction.
  • Scaling and Domain Performance: Large models outperform smaller models, but gains become less pronounced beyond about 10B parameters.Large models average 0.3009, compared with 0.1460 for Medium and 0.0854 for Small; all pairwise differences are significant.
  • Scaling and Domain Performance: Performance ranges from 0.1179 to 0.332 across 13 domains, with Health/Medical highest and Entertainment/Arts and Gaming most challenging.Large models retain positive scale effects in every domain, including the lowest-performing domains.

E.5 Visual-Text Grounding Error Analysis

Explicitation increases labeled visual-text grounding errors largely by unmasking failures that were previously classified under other categories. Error tracking and examples show that models move from vague responses to specific but sometimes incorrect visual interpretations.

  • Error tracking: Explicitation raised VTG errors from 5.2% to 16.6%, prompting tracking of individual question-model pairs across conditions.The tracking distinguishes newly surfaced errors from persistent failures and reclassified cases.
  • Error tracking: 87.0% of 461 explicitated-condition VTG errors were newly surfaced, while 13.0% persisted across both conditions.Additionally, 104 VTG errors from the original condition were resolved by explicitation.
  • Interpretation: VTG error severity remained nearly unchanged, with severe errors declining from 84.8% to 83.7% after explicitation.This pattern supports interpreting the increase as error unmasking rather than a broad change in severity.
  • Example: In one example, a vague original response became a specific visual-text error when the model misread 惹 as 芯 after explicitation.The same question therefore received different error labels because the explicit query directed attention to a specific visual region.
  • Error annotation: Errors were annotated by Claude 3.5 Sonnet using the original question, gold answer, checklist, model response, and metadata under a predefined taxonomy.The annotation schema assigns a root cause, one to three failure categories, and a severity level.
Loading 2601.06165v2…