PaperlayerView exact revision on arXiv

Source-linked AI summary

V-REX: Benchmarking Exploratory Visual Reasoning via Chain-of-Questions

Chenrui Fan, Yijun Liang, Shweta Bhardwaj, Kwesi Cobbina, Ming Li, Tianyi Zhou

arXiv:2512.11995v2cs.CVcs.AIcs.LG

TL;DR

VLMs struggle with open-ended visual tasks that require progressive exploration, while evaluating their intermediate reasoning is difficult. V-REX uses a Chain-of-Questions benchmark to assess Planning and Following separately, revealing scaling trends and persistent differences between these abilities.

  • Problem

    Open-ended visual tasks require progressive exploration, but existing VLMs often struggle to plan adaptively and these intermediate reasoning paths are difficult to evaluate.

    Source: §1 · pp. 1–2

  • Method

    V-REX structures exploratory reasoning as a finite-choice Chain-of-Questions benchmark that separately evaluates selecting subquestions (Planning) and answering them (Following).

    Source: §1 · pp. 3–4, §3.1 · p. 5

  • Results

    V-REX reveals scaling trends, better Following than Planning in smaller models, and better final-question performance when models follow CoQ hints.

    Source: §1 · p. 4

  • Takeaways & Limitations

    V-REX identifies effective exploratory planning as a key challenge for future VLMs and provides a benchmark for studying stepwise visual reasoning.

    Source: §6 · p. 14

    For some models, CoQ steps can distract reasoning or omit essential cues, disrupting question or answer selection.

    Source: §5.2 · p. 11

Abstract

from arXiv · show

While many vision-language models (VLMs) are developed to answer well-defined, straightforward questions with highly specified targets, as in most benchmarks, they often struggle in practice with complex open-ended tasks, which usually require multiple rounds of exploration and reasoning in the visual space. Such visual thinking paths not only provide step-by-step exploration and verification as an AI detective but also produce better interpretations of the final answers. However, these paths are challenging to evaluate due to the large exploration space of intermediate steps. To bridge the gap, we develop an evaluation suite, ``Visual Reasoning with multi-step EXploration (V-REX)'', which is composed of a benchmark of challenging visual reasoning tasks requiring native multi-step exploration and an evaluation protocol. V-REX covers rich application scenarios across diverse domains. V-REX casts the multi-step exploratory reasoning into a Chain-of-Questions (CoQ) and disentangles VLMs' capability to (1) Planning: breaking down an open-ended task by selecting a chain of exploratory questions; and (2) Following: answering curated CoQ sequentially to collect information for deriving the final answer. By curating finite options of questions and answers per step, V-REX achieves a reliable quantitative and fine-grained analysis of the intermediate steps. By assessing SOTA proprietary and open-sourced VLMs, we reveal consistent scaling trends, significant differences between planning and following abilities, and substantial room for improvement in multi-step exploratory reasoning.

1 Introduction

V-REX addresses the difficulty of evaluating open-ended visual reasoning that requires active, multi-step exploration. It uses Chain-of-Questions to assess planning and following separately and reports scaling trends alongside differences between those abilities.

1 Introduction

  • Practical VLM applications include tasks whose goals are not fully specified at the outset and require multiple rounds of exploration and sub-goal proposal.Examples include guessing a location from street-view imagery and detecting cheating from posted images.Source: §1 · pp. 1–2
  • Existing benchmarks often use specified, single-round questions or toy settings and score only final answers, leaving intermediate steps and recovery from errors unclear.Open-ended tasks instead require active clue gathering and progressive planning; poor exploration may undermine or distract reasoning.Source: §1 · pp. 2–3
  • V-REX formulates exploratory reasoning as interconnected sub-questions and answers, then evaluates question selection (Planning) and sub-question answering (Following) separately.CoQ uses finite choices per step to make exploration tractable and controllable.Source: §1 · pp. 3–4
  • 702 samples and 2,504 questions span 4 reasoning categories and 15 scenarios, with 2–6 steps per sample and 3.57 steps on average.The scenarios include settings such as diagrams, time estimation, and GUI interpretation.Source: Figure 3 · p. 3, §1 · p. 4
  • Following CoQ hints improves final-question performance; models scale with size, smaller models favor Following over Planning, and Planning varies more across models of the same size.Following and Planning both contribute positively to overall performance, and VLMs recover less well from failed Following than from failed Planning.Source: §1 · p. 4

2 Related Work

Prior visual reasoning benchmarks cover varied tasks but often neglect question-space exploration, while evaluation of intermediate reasoning has used methods V-REX identifies as unreliable or judgment-dependent.

2.1 Exploratory Visual Reasoning

  • Prior visual reasoning benchmarks cover many task types but largely neglect question-space exploration; V-REX evaluates helpful sub-question proposal alongside answer-space exploration.The related work lists cognitive reasoning, visual grid reasoning [33], comparison [3], cultural reasoning, multi-image reasoning [9], and color-related reasoning [21].Source: §2.1 · pp. 4–5

2.2 Evaluation with Intermediate Steps

  • Most benchmarks measure final-answer accuracy, while Chen et al. [5] use Monte-Carlo estimation and LLM-as-a-Judge for process rewards and LlamaV-o1 uses LLM judgments of trace alignment or correctness.V-REX instead breaks tasks into sub-tasks and evaluates Planning and Following.Source: §2.2 · p. 5

3 Exploratory Reasoning in Visual Space

The Chain-of-Questions formulation represents exploratory reasoning as linked question-answer steps and evaluates question selection separately from answering a supplied chain.

3.1 Chain-of-Questions (CoQ)

  • CoQ represents exploratory reasoning as a sequence of interconnected sub-questions and answers, evaluated through finite-space multiple-choice decisions.This formulation separates informative intermediate-question generation (Planning) from answering those questions accurately (Following).Source: §3.1 · p. 5

3.2 Planning of CoQ

  • Planning tests whether a model selects an informative next sub-question given the original problem and prior CoQ history.At each step, distractor questions are added to the ground-truth chain, and the model receives the ground-truth answer so it need not answer the sub-question itself.Source: §3.2 · p. 6

3.3 Following of CoQ

  • Following tests whether a model answers each ground-truth sub-question from image cues and accumulated conversation history, choosing among answer distractors.The model follows a pre-constructed sequential QA chain toward the final conclusion.Source: §3.3 · p. 6

4 V-REX Benchmark

V-REX provides a benchmark for multi-step exploratory visual reasoning, organized across diverse categories and scenarios. It uses QA chains to curate tasks and separately evaluate Planning, Following, and intermediate accuracy.

4 V-REX Benchmark

  • V-REX contains 702 samples and 2,504 questions across four reasoning categories and 15 scenarios, with 2–6 steps per sample (3.57 on average).Figure 4 presents question and reasoning-step distributions.Source: §4 · p. 6, Figure 4 · p. 7

4.1 Taxonomy

  • The taxonomy comprises Deduction (4 scenarios), Guessing (5), Navigation (4), and Retrieval (2), covering distinct reasoning approaches.Deduction covers logical or causal relationships; Guessing explores hidden factors under uncertainty; Navigation traverses spatial or procedural paths; Retrieval gathers dispersed visual information.Source: §4.1 · p. 7

Ground-Truth QA Chain

  • Ground-truth QA chains are sequences of question-answer pairs ending in a final question and answer, with intermediate questions designed to be helpful and correctly ordered.Each question should help answer the following questions, and earlier questions should not depend on answers to later questions.Source: Ground-Truth QA Chain · pp. 7–8

4.2 Data Curation

  • Five PhD-level annotators initially constructed ground-truth chains for assigned scenarios, followed by two cross-verification rounds checking validity, helpfulness, and step order.In each round, annotators were shuffled to three other unseen scenarios and gave feedback to the original annotator and previous verifier.Source: Ground-Truth QA Chain · p. 8

Planning Task

  • Planning samples use LLM-assisted distractor construction and question integration, with locally plausible step-level and globally self-consistent chain-level misleading alternatives.The integrated dataset is automatically filtered and refined.Source: Planning Task · p. 8

Following Task

  • Following samples turn manually created plausible distractors and the ground-truth answer at each step into multiple-choice questions.Source: Following Task · p. 8

4.3 Evaluation Metrics

  • Intermediate accuracy evaluates both dimensions: Planning compares chosen questions with ground-truth questions, while Following evaluates answers to ground-truth questions.Source: §4.3 · p. 9

5 Results

V-REX results show scaling across Following and Planning, alongside distinct strengths between model families and substantial differences in how consistently models perform. CoQ hints generally improve final answers, and both abilities correlate with end-to-end performance, while smaller models favor Following and recovery differs by error type.

5.1 Main Results

  • InternVL2.5-38B leads in 3 of 4 Following categories and InternVL3-38B in 3 Planning categories, while expertise varies more among smaller models.Proprietary models such as GPT-5 and o3 also achieve top accuracy across multiple categories.Source: §5.1 · p. 9
  • GPT-5 and o3 achieve the highest average performance on both tasks, while the gap between large open-source and proprietary models has narrowed considerably.Several large open-source models match or exceed proprietary models in Deduction and Navigation.Source: §5.1 · p. 9, Table 1 · p. 10

5.2 Main Findings

  • Most CoQ performance-change ratios are positive; Retrieval gains are relatively minor, and a few models decline when hints introduce distracting paths or omit cues.The change ratio compares final accuracy with and without CoQ hints across models and reasoning categories.Source: §5.2 · pp. 10–11, Figure 5 · p. 11
  • Ability rises with model size, while same-sized models show less variance in Following than in Planning.The size relationship is positive for both tasks; the smaller Following variance is described across model families.Source: §5.2 · p. 11, Figure 6 · p. 12
  • Following and Planning both correlate positively with end-to-end performance, with Pearson correlations of 0.948 and 0.858, respectively.Following remains a primary determinant of end-to-end capability, while Planning varies more across VLMs.Source: Figure 7 · p. 12, §5.2 · pp. 12–13
  • Smaller models perform better at Following than Planning, whereas larger models show more balanced abilities as the ratio approaches unity.The Following-to-Planning ratio becomes more balanced as model size increases.Source: Figure 8 · p. 13, §5.2 · p. 13

5.3 Additional Results

  • Additional analyses include a blindfold baseline, increased distractor complexity, and fine-grained case studies of the benchmark.These complementary results and experimental details are provided in the appendices.Source: §5.3 · p. 14

5.2 Main Findings

  • Models above 10B parameters show some failure recovery, generally recovering more often from failed Planning than failed Following; larger proprietary models are stronger on Planning recovery.Recovery means reaching the correct final answer despite at least one incorrect intermediate step.Source: §5.2 · pp. 13–14, Table 2 · p. 14

6 Conclusion

V-REX evaluates exploratory visual reasoning through Chain-of-Questions and reports scaling trends, links between intermediate abilities and final performance, and a Following-over-Planning imbalance in smaller models. Its 15-scenario taxonomy groups exploratory tasks across four reasoning categories.

6 Conclusion

  • Experiments reveal scaling trends, strong correlations between Planning and Following abilities and final performance, and a smaller-model imbalance favoring Following over Planning.The authors identify effective planning as a key challenge for future VLMs.Source: §6 · p. 14

A Image Sources

  • Images combine benchmark sources PhysBench[10], MathVision[41], CV-Bench[40], and InfographicQA[25] with GeoGuessr[26] and TimeGuessr[12], covering 15 scenarios.Source: A Image Sources · p. 20

B Detailed Taxonomy

  • Deduction tasks probe discovering and applying rules, while Guessing tasks assess resolving uncertainty through hypotheses and contextual evidence.Examples include tracing flowcharts, inferring properties, and guessing locations or intentions.Source: B Detailed Taxonomy · pp. 20–21
  • Navigation evaluates planning and adjusting spatial or sequential trajectories; Retrieval assesses locating and integrating dispersed visual or textual evidence.The taxonomy includes map, GUI, traffic, and trend navigation, alongside counting and word-puzzle retrieval.Source: B Detailed Taxonomy · pp. 20–21
  • Together, the four categories and 15 subtasks capture complementary ways VLMs hypothesize, search, plan, and integrate knowledge across multimodal scenarios.Source: B Detailed Taxonomy · p. 21

C Data Curation for Planning

Planning distractors are generated and selected to test whether VLMs choose useful questions among plausible alternatives. The evaluation separately prompts Planning and Following and compares final-question accuracy across CoQ settings.

C.1 Generation Process

  • A two-stage pipeline constructs distractors and integrates selected alternatives into the Planning dataset, combining LLM generation, filtering, and refinement.It retains a challenging chain-level distractor and diverse step-level alternatives for each sample.Source: §C.1 · p. 22

C.2 Generation Prompt

  • Generation prompts require visually grounded, coherent distractors that appear plausible but are strategically irrelevant and avoid clues useful for the final answer.Separate prompts target local step-level alternatives and globally coherent distracting chains.Source: §C.2 · pp. 23, 26, 27

D.1 Implementation Details

  • The evaluation covers 32 proprietary and open-source models, with open-source systems ranging from 1B to 38B parameters and tested under standardized conditions.Open-source models use one NVIDIA A100 GPU (80GB) with identical decoding and temperature settings.Source: §D.1 · p. 28

D.2 Evaluation Prompts

  • Planning prompts models to select a helpful intermediate question from candidates, while Following asks them to select the correct answer along the curated chain.Source: §D.2 · p. 28

E Final Accuracy for VLMs on V-REX

  • Table 3 compares final-question accuracy without CoQ and under Planning or Following, showing performance changes across the three evaluation modes.The modes assess direct answering, question-space exploration, and answering along the ground-truth chain.Source: E Final Accuracy for VLMs on V-REX · p. 29

F Blindfold Experiment on V-REX

Without images, performance drops substantially on most V-REX categories, but Navigation is a notable exception.

F Blindfold Experiment on V-REX

  • Without visual input, performance drops significantly for Guessing, Deduction, and Retrieval, with most models near random guessing on these tasks.Scores are around 25% for four-option questions, or lower depending on task format.Source: F Blindfold Experiment on V-REX · p. 29
  • Navigation scores remain higher without images, suggesting textual spatial cues or commonsense patterns can contribute substantially in that category.The authors describe visual information as potentially supplementary rather than the sole critical evidence for Navigation decisions.Source: F Blindfold Experiment on V-REX · p. 30

G Exploring Task Difficulty on Planning

Expanding the Planning candidate set makes the task harder, while model rankings remain largely stable.

G Exploring Task Difficulty on Planning

  • Performance consistently decreases across models of different sizes and architectures when four distracting options are added to each intermediate question.The larger candidate set increases the challenge of selecting the correct reasoning step among plausible alternatives.Source: G Exploring Task Difficulty on Planning · pp. 30–31
  • Despite lower absolute performance with more options, model rankings remain largely consistent, suggesting the smaller candidate set approximates comparative exploration ability.The expanded set mainly increases difficulty without significantly changing relative ordering.Source: G Exploring Task Difficulty on Planning · p. 31

H Stepwise Recovery from Failure Analysis

Wrong Following steps are associated with steadily lower final accuracy, while failed Planning shows a different, non-monotonic pattern. Expanding the candidate options also lowers Planning performance across models.

H Stepwise Recovery from Failure Analysis

  • Final accuracy decreases monotonically as the number of wrong Following steps increases, whereas accuracy after failed Planning is not monotonic.The authors hypothesize that many wrong Planning steps may lead a model to take a different approach and still reach the correct conclusion.Source: H Stepwise Recovery from Failure… · p. 31
  • Models are more robust to wrong Following steps than wrong Planning steps, with final accuracy dropping more sharply after Following errors.Source: H Stepwise Recovery from Failure… · p. 31
  • Expanding intermediate-question options from 4 to 8 consistently lowers Planning performance across models, while relative model rankings remain largely stable.Source: Table 5 · p. 32

I Shared Failures Analysis among Models

Step-wise shared failures cluster at particular positions in hard samples, while moderate samples rarely show near-universal failures. Difficulty labels based on intermediate steps reveal systematic bottlenecks that average final-answer accuracy can obscure, and CoQ decomposition can help models correct perceptual mistakes or stay on the intended reasoning path.

I Shared Failures Analysis among Models

  • Final-answer averages can mask intermediate failures, and recovery or shortcuts can yield high accuracy despite process-level errors.Source: I Shared Failures Analysis among Models · p. 32
  • Difficulty is assigned from intermediate-step performance across six model families: easy samples have perfect average step accuracy, while hard samples have a unanimous failure or average accuracy below 0.5.Following has 69 hard, 224 moderate, and 82 easy samples; Planning has 43 hard, 287 moderate, and 21 easy samples.Source: I Shared Failures Analysis among Models · pp. 32–33
  • Hard samples show step positions where at least 5 of 6 models fail, while moderate samples rarely do, with bottleneck positions varying by task category.The analysis covers both Following and Planning; the shared-failure fraction is reported by step for hard and moderate samples.Source: I Shared Failures Analysis among Models · pp. 33–34, Figure 12 · p. 34
  • Shared failures peak earlier in Retrieval, Navigation, and Deduction, but later in Guessing, pointing to bottlenecks in both grounding and later integration or decision steps.The authors conclude that these shared step-level failures leave benchmark headroom even when average accuracy is high.Source: I Shared Failures Analysis among Models · p. 34

J.1 Success cases

  • Decomposing the final question into sub-questions can help a model correct perceptual mistakes or stay aligned with the intended reasoning path.Source: §J.1 · p. 34

J.2 Failure cases of CoQ

CoQ can offer limited benefit on Retrieval tasks that rely on factual lookup or direct visual identification, and its prescribed chain can conflict with a model’s natural retrieval strategy. Planning may also introduce distracting side paths, while Following can fail through either an incorrect intermediate answer or failure to use information already collected.

J.2 Failure cases of CoQ

  • CoQ offers limited benefit for Retrieval problems centered on factual lookup or direct visual identification rather than layered reasoning.Adding a chain may contribute little when the task involves minimal reasoning depth.Source: §J.2 · pp. 34–35
  • On some dense-image Retrieval cases, the model can answer correctly without CoQ but fail with it because fine-grained perception, not exploration, is the bottleneck.Source: §J.2 · p. 35, Figure 16 · p. 37
  • A human-designed CoQ chain can conflict with a model’s natural retrieval process, leading to incorrect question selection and a wrong final answer even under Following.Source: §J.2 · p. 35, Figure 17 · p. 38

J.3 Failure cases of Planning

  • In Planning, visually salient but semantically irrelevant steps can distract models from needed cues and disrupt both step selection and the final prediction.Examples include an irrelevant question about birds and a chain focused on color descriptions rather than the intended concept.Source: §J.3 · pp. 35–36, Figure 18 · p. 39, Figure 19 · p. 39

J.4 Failure cases of Following

Loading 2512.11995v2…