PaperlayerView exact revision on arXiv

Source-linked AI summary

PhysVLM-AVR: Active Visual Reasoning for Multimodal Large Language Models in Physical Environments

Weijie Zhou, Xuantang Xiong, Yi Peng, Manli Tao, Chaoyang Zhao, Honghui Dong, Ming Tang, Jinqiao Wang

arXiv:2510.21111v1cs.CV

TL;DR

Static visual reasoning is limited when physical environments hide relevant information, while AVR makes interaction part of reasoning. The paper introduces an interactive benchmark, a structured dataset, and PhysVLM-AVR; the model achieves state-of-the-art performance on CLEVR-AVR and shows strong generalization to other embodied and static reasoning tasks.

  • Problem

    Static, fully observable visual inputs leave MLLMs without needed information in physical environments with occlusion or limited views.

    Source: §1 · p. 2

  • Method

    The paper defines AVR as sequential interaction and multi-step reasoning, and develops CLEVR-AVR, AVR-152k with AVR-Core CoT annotations, and PhysVLM-AVR.

    Source: §1 · pp. 2–3

  • Results

    PhysVLM-AVR achieves state-of-the-art performance on CLEVR-AVR and shows strong generalization to other embodied and static reasoning tasks.

    Source: §7 · p. 10

  • Takeaways & Limitations

    Current embodied MLLMs can identify information incompleteness but still struggle to strategically acquire and integrate information through interaction.

    Source: §6.2 · p. 8, §7 · p. 10

    The authors identify optimal action selection and multi-step information integration as remaining challenges, despite models’ high information-sufficiency judgment accuracy.

    Source: §6.2 · p. 8

Abstract

from arXiv · show

Visual reasoning in multimodal large language models (MLLMs) has primarily been studied in static, fully observable settings, limiting their effectiveness in real-world environments where information is often incomplete due to occlusion or limited field of view. Humans, in contrast, actively explore and interact with their environment-moving, examining, and manipulating objects-to gather information through a closed-loop process integrating perception, reasoning, and action. Inspired by this human capability, we introduce the Active Visual Reasoning (AVR) task, extending visual reasoning to partially observable, interactive environments. AVR necessitates agents to: (1) actively acquire information via sequential physical actions, (2) integrate observations across multiple steps for coherent reasoning, and (3) dynamically adjust decisions based on evolving visual feedback. To rigorously evaluate AVR, we introduce CLEVR-AVR, a simulation benchmark featuring multi-round interactive environments designed to assess both reasoning correctness and information-gathering efficiency. We present AVR-152k, a large-scale dataset that offers rich Chain-of-Thought (CoT) annotations detailing iterative reasoning for uncertainty identification, action-conditioned information gain prediction, and information-maximizing action selection, crucial for training agents in a higher-order Markov Decision Process. Building on this, we develop PhysVLM-AVR, an MLLM achieving state-of-the-art performance on CLEVR-AVR, embodied reasoning (OpenEQA, RoboVQA), and passive visual reasoning (GeoMath, Geometry30K). Our analysis also reveals that current embodied MLLMs, despite detecting information incompleteness, struggle to actively acquire and integrate new information through interaction, highlighting a fundamental gap in active reasoning capabilities.

1 Introduction

Static visual reasoning leaves MLLMs poorly matched to partially observable environments, where occlusion and limited views hide information needed for answers. The paper introduces AVR and develops a benchmark, dataset, and model around sequential information gathering and reasoning.

1 Introduction

  • Static, passive visual inputs limit MLLMs in physical environments where occlusion, containment, or limited field of view leave relevant information incomplete.The introduction describes passive models answering incorrectly from an initial partial view, while interaction can reveal occluded objects.Source: §1 · p. 2
  • Existing visual reasoning, embodied question answering, and exploration frameworks often fail to link strategic sequential actions with reasoning updates as new information arrives.The paper contrasts complete-input, single-step reasoning; EQA from video sequences; and exploration optimized for task completion rather than explicit information gain.Source: §1 · p. 2
  • AVR extends visual reasoning to partially observable, interactive environments, requiring sequential information gathering, multi-step observation integration, and decisions that adapt to visual feedback.The paradigm links perception, reasoning, and action to resolve uncertainty.Source: §1 · p. 2
  • CLEVR-AVR evaluates reasoning correctness and information-gathering efficiency in multi-round interactive environments.AVR-152k supplies multi-level training annotations, including uncertainty identification, action-conditioned information-gain prediction, and information-maximizing action selection.Source: §1 · p. 3
  • The paper presents PhysVLM-AVR as state of the art on CLEVR-AVR, with strong results on standard embodied visual reasoning tasks, and identifies limits in current MLLMs’ active interaction.The authors frame the work as a foundation for MLLMs that actively reason about and interact with physical environments.Source: §1 · p. 3

2 Related Work

Related work addresses visual reasoning, embodied question answering, and embodied exploration, but the paper argues these lines do not fully connect reasoning needs to active information gathering. AVR is presented as combining goal-directed interaction with multi-step visual reasoning.

Visual Reasoning

  • Traditional visual reasoning tasks use static, fully observable images and do not model the sequential interaction needed to acquire information in partially observable scenarios.AVR instead requires sequential actions to obtain information needed for reasoning.Source: §2 · p. 3

Embodied Question Answering

  • Embodied question answering often relies on pre-captured data, underemphasizing the closed loop in which reasoning drives information-seeking actions and new observations refine understanding.AVR centers on this loop and integrates observations acquired through goal-directed actions.Source: §2 · p. 3

Embodied Exploration

  • Embodied exploration methods typically optimize task success rather than information acquisition targeted to a reasoning objective.AVR prioritizes uncertainty-driven actions that directly maximize goal-relevant information gain.Source: §2 · p. 3

Summary

  • AVR integrates interaction and reasoning by making active perception responsive to reasoning needs in partially observable environments.The paper relates this design to active perception as a feature of human cognition.Source: §2 · p. 3

3 Active Visual Reasoning

The paper formulates Active Visual Reasoning as a closed-loop process in which agents reason over accumulated observations and either answer or act to gain information. It identifies efficient exploration, temporal integration, and reasoning-driven action as core challenges.

3.1 Problem Formulation

  • At each timestep, the agent uses the question and observation history to reason, answers if the history is sufficient, or selects an information-gathering action otherwise.The action produces a new observation that updates the history, and the cycle continues until an answer or the timestep limit.Source: §3.1 · pp. 3–4

3.2 Key Challenges

  • AVR requires efficient exploration of partial views, temporal reasoning over multi-step observations, and actions guided by current understanding to reduce uncertainty.These challenges motivate dedicated benchmarks and datasets that capture sequential decision-making.Source: §3.2 · p. 4

4 CLEVR-AVR Benchmark and AVR-152k Dataset

CLEVR-AVR evaluates active reasoning through interactive simulation, while AVR-152k provides structured data for learning the reasoning and decisions in that loop. AVR-Core models each interaction step in a higher-order MDP and includes expert-structured reasoning annotations.

4.1 CLEVR-AVR: A Benchmark for Evaluating AVR Capabilities

  • CLEVR-AVR is a Genesis-based interactive simulation extending CLEVR to assess the perception-reasoning-action loop, with varied occlusion and stacking scenarios and actions for object manipulation or viewpoint changes.Questions require at least two reasoning steps on average, with some scenarios requiring 4–6 interaction steps.Source: §4.1 · p. 4
  • CLEVR-AVR measures information-sufficiency judgment, information-gain rate, and final-answer accuracy; information gain is determined by whether an action reveals previously occluded or stacked question-relevant content.The benchmark’s question options let agents choose a final answer or an intermediate action, supporting uncertainty-reducing interaction rather than random exploration.Source: §4.1 · pp. 4–5

4.2 AVR-152k Dataset: Modeling Active Visual Reasoning as a Higher-Order MDP

  • AVR-152k introduces AVR-Core as a higher-order MDP framework for learning sequential information gathering and reasoning in interactive environments.Each sample includes the question, observation and action histories, available actions, a reasoning trace, and either a selected action or final answer.Source: §4.2 · pp. 5–7
  • AVR-Core CoT annotations assess current understanding, predict which actions could resolve uncertainty, and choose an action or final answer when uncertainty is resolved.The annotations are designed to represent a multi-step information-seeking process.Source: §4.2 · p. 5
  • AVR-152k comprises AVR-Caption (100k samples) for visual perception and spatial understanding, AVR-Embodied Reasoning (50k samples) for temporal visual reasoning, and AVR-Core (2k samples), instantiating AVR’s higher-order MDP.Source: §4.2 · p. 6
  • AVR-Core annotations were initially authored by human experts, refined by Gemini, and validated through expert annotation, logical-consistency checks, and human review.The review covers perception-reasoning-action validity.Source: §4.2 · p. 6

5 PhysVLM-AVR Model

PhysVLM-AVR pairs a multi-image architecture with multi-stage, mixed-data training to build active visual reasoning capabilities. Training progresses from visual-language alignment through broader understanding to general and active reasoning.

5 PhysVLM-AVR Model

  • Max pooling after the visual encoder reduces visual tokens threefold, enabling more efficient processing of multiple images.The architecture also uses a Qwen2.5-3B language-model decoder and a SigLIP-400M visual encoder.Source: §5 · p. 7
  • Stage 1 aligns visual features with the language model by training only the connector layers on LLaVA-Pretrain.Source: §5 · p. 7
  • Stage 2.1 builds foundational image comprehension by fine-tuning all parameters on LLaVA-OneVision-data.Source: §5 · p. 7
  • Stage 2.2 broadens visual understanding through training on M4-Instruct-data and AVR-Caption.Source: §5 · p. 7
  • Stage 3 combines general reasoning and active visual reasoning training using Reason-RFT-129k, AM-DeepSeek-R1-Distilled-100k, AVR-Embodied Reasoning, and AVR-Core.Source: §5 · p. 7
  • The multi-stage strategy culminates in PhysVLM-AVR-3B; its detailed training configuration is provided in Appendix Section C and Figure A-13.Source: §5 · p. 7

6 Experiment

The experiments evaluate AVR models on active, embodied, and static visual reasoning tasks, with ablations testing the roles of AVR-Core and its CoT annotations. Results report strong performance across benchmarks, alongside a remaining gap between recognizing insufficient information and reasoning to a final answer.

6.1 Experimental Setup

  • Evaluation covers CLEVR-AVR with ACCISJ, IGR, and ACCFA; OpenEQA with LLM-score; RoboVQA with BLEU1-4; and GeoMath and Geometry3K.The comparisons include open-source, visual-reasoning, embodied, and API-based MLLMs.Source: §6.1 · pp. 7–8

6.2 Results on Active Visual Reasoning Tasks

  • On CLEVR-AVR, existing open-source and passive visual-reasoning models perform near zero, while Embodied-Reasoner-7B scores 20.2% ACCISJ but only 1.6% ACCFA.The authors identify a gap between detecting incomplete information and strategically acquiring and integrating it; even PhysVLM-AVR-3B’s 90.5% ACCISJ exceeds its 39.7% ACCFA.Source: §6.2 · p. 8

6.2–6.4 Results

  • PhysVLM-AVR-3B records 39.7% ACCFA on CLEVR-AVR, while the proposed models outperform embodied baselines and PhysVLM-AVR-3B leads on GeoMath and Geometry3K.The embodied evaluation includes OpenEQA and RoboVQA; the models’ RoboVQA scores are close to or exceed fully supervised baselines despite using 1/20 of the training set.Source: §6.2 · p. 8, §6.3 · p. 9, §6.4 · p. 9

6.5 Ablation Study

  • Removing AVR-Core drops ACCISJ from 90.5% to 16.4%, ACCFA from 39.1% to 2.3%, and IGR from 29.9% to 11.2%; removing CoT yields 47.6% ACCISJ, 18.0% IGR, and 16.9% ACCFA.The annotations supervise uncertainty identification, action-conditioned information-gain prediction, and strategic action selection.Source: §6.5 · p. 9, Table 2 · p. 9

7 Conclusion

The paper introduces Active Visual Reasoning (AVR) for interactive, partially observable environments and develops resources and a model for the task. PhysVLM-AVR shows strong results, while strategic information gathering and integration remain challenges.

7 Conclusion

  • PhysVLM-AVR achieves state-of-the-art performance on CLEVR-AVR and shows strong generalization to other embodied and static reasoning tasks.The conclusion identifies OpenEQA, RoboVQA, GeoMath, and Geometry30K as examples of those other tasks.Source: §7 · p. 10
  • The paper introduces AVR and develops CLEVR-AVR and AVR-152k, whose AVR-Core component provides detailed CoT annotations in a higher-order MDP framework for training agents.Source: §7 · pp. 9–10
  • Current models can identify incomplete information, but choosing information-gathering actions and integrating new information remain challenges.Future work will focus on predicting action-conditioned information gain, selecting information-gathering actions, and exploring more complex real-world scenarios and sample efficiency.Source: §7 · p. 10

A More details of CLEVR-AVR benchmark

The appendix figures document CLEVR-AVR's scene types, question templates, and distributions of occlusion and stacking across scene types.

A More details of CLEVR-AVR benchmark

A.1 Action Candidate Generation in CLEVR-AVR

CLEVR-AVR combines a fixed set of physical actions with varied questions and scene-dependent candidate actions, including distractors that test information-seeking choices.

A.1 Action Candidate Generation in CLEVR-AVR

  • Agents use a Franka robotic manipulator and camera controls to perform Pick, Move Viewer, Rotate Viewer, and Move Object actions.Source: §A.1 · p. 15
  • The benchmark includes Query, Exist, Counting, Compare, Math Counting, and Math Compare question types.Source: §A.1 · p. 15
  • Each step presents 5–8 candidate actions, with approximately 3–5 designed as distractors or sub-optimal choices that offer less information gain.Targets for object actions are selected dynamically from the current visual scene, often to resolve occlusions or ambiguities.Source: §A.1 · p. 15

B More details of AVR-152k dataset

The appendix documents AVR-152k data examples and describes how human experts generated AVR-Core reasoning annotations through task execution and post-interaction refinement.

B More details of AVR-152k dataset

  • Figures A-4–A-8 show AVR-Caption and AVR-Embodied Reasoning prompts and examples, while Figures A-9–A-12 cover AVR-Core refinement prompts and a sequential instance.Source: B More details of AVR-152k dataset · p. 15

B.1 Initial Generation of Expert CoT Annotations for AVR-Core

  • Experts performed interactive tasks while logging reasoning, uncertainties, hidden-information hypotheses, and intended-action rationales, then structured those logs using the complete interaction record.This human-centric process aimed to ground CoTs in plausible and effective information-gathering strategies for later refinement and training.Source: §B.1 · pp. 15–16
  • The illustrated AVR-Core sequence moves from recognizing insufficient information and planning actions, through reassessment after observations, to answering once sufficient evidence is gathered.Source: §B.1 · p. 16

C More details of Model and Training

The appendix documents model and training details, the prompt used for CLEVR-AVR baselines, and supplementary benchmark, data, and usage materials. These include staged examples of active reasoning and restrictions on research-resource use.

C More details of Model and Training

D Baseline Model Input Prompt for CLEVR-AVR Experiments

  • CLEVR-AVR baseline prompts instruct models to act when observations are insufficient, gather relevant information, and answer from accumulated visual history.The prompt permits choosing either a final answer or an action option.Source: D Baseline Model Input Prompt for… · p. 16

F Ethical Considerations and Usage Restrictions

NeurIPS Paper Checklist

The checklist records author responses about claim scope, reproducibility, experimental reporting, limitations, and research ethics, alongside guidance on what those disclosures should address.

Claims and scope

Limitations

  • The authors report discussing the work’s limitations in the Appendix.Checklist guidance calls for addressing assumptions, robustness, claim scope, and factors affecting performance.Source: NeurIPS Paper Checklist · p. 28

Reproducibility and access

  • The authors say experimental details needed to reproduce the main results are disclosed in the Experiment section and Appendix.They also report using anonymous code links in accordance with submission guidelines.Source: NeurIPS Paper Checklist · pp. 29–30

Experimental details

  • The checklist asks for training and test details, compute resources, and appropriate statistical-significance reporting; the authors refer to the Experiment section and Appendix for details.They report averaging experiments multiple times and setting random seeds to 42.Source: NeurIPS Paper Checklist · pp. 30–31

Ethics and societal impacts

  • The authors state that their research complies with the NeurIPS Code of Ethics and that societal impacts are discussed in the Appendix.Source: NeurIPS Paper Checklist · p. 31

Assets and human subjects

  • The checklist covers crediting existing assets, documenting new assets, and reporting on human-subject research; the authors say they do not release new assets or conduct human-subject studies.They report referencing relevant code and models while respecting their licenses.Source: NeurIPS Paper Checklist · pp. 32–33
Loading 2510.21111v1…