PaperlayerView exact revision on arXiv

Source-linked AI summary

JRDB-AVR: An Active Visual Reasoning Benchmark for Embodied Agents in Real-World Environments

Zhixi Cai, Fucai Ke, Sukai Huang, Maria Garcia de la Banda, Peter J. Stuckey, Gholamreza Haffari, Hamid Rezatofighi

arXiv:2609.35032v2cs.AIcs.CVcs.RO

TL;DR

Embodied visual reasoning can require evidence from times, viewpoints, and objects that are not all visible at once, while many benchmarks use passive observations and final-answer scoring. JRDB-AVR evaluates active observation and grounded evidence, and JRDB-AVR-Agent provides a graph-based reference method. Results show that answer and evidence performance can diverge, and the agent improves evidence grounding over VLM baselines.

  • Problem

    Embodied visual reasoning may require evidence across time, viewpoint, and interacting objects, but current benchmarks largely evaluate passive observations and final answers.

    Source: §1 · pp. 1–2

  • Method

    JRDB-AVR derives active-reasoning questions from JRDB and evaluates timestamp-and-view observation requests alongside answers and grounded visual evidence.

    Source: §1 · pp. 2–3

  • Results

    JRDB-AVR-Agent improves evidence grounding over VLM baselines, while the remaining gap highlights the difficulty of active observation and evidence grounding.

    Source: §6 · p. 10

  • Takeaways & Limitations

    Reporting answer, evidence, and combined correctness exposes unsupported answers that answer-only evaluation would hide.

    Source: §1 · p. 3, §6 · p. 10

    JRDB-AVR evaluates active reasoning over recorded observations rather than live robot control.

    Source: §6 · p. 10

Abstract

from arXiv · show

In complex embodied visual reasoning scenarios, an agent often has only a limited field of view, and the evidence needed to answer a question may be distributed across time, viewpoint, and interacting objects. A model may therefore give a plausible answer without ever observing the relevant object, time, or view that supports it. Current visual reasoning benchmarks largely evaluate passive observations and final answers, overlooking settings that require active reasoning and evidence acquisition. We introduce JRDB-AVR, a benchmark derived from existing real-world JRDB robotics data through a structured question-generation engine that turns this gap into an explicit evaluation: an embodied agentic system receives a visual reasoning question, requests bounded observations by timestamp and viewing angle, and is evaluated on both the final answer and the grounded visual evidence supporting it. The benchmark contains diverse questions over multiple real-world environments involving temporal search, viewpoint selection, and human-oriented compositional reasoning. We also introduce JRDB-AVR-Agent, a reference active reasoning agentic method that maintains an explicit observation-grounded graph-based world model and answers through solving. Experiments reveal a substantial gap between answer accuracy and evidence accuracy in current baselines, showing that current VLMs can produce unsupported correct answers and that active evidence-aware evaluation is necessary for embodied visual reasoning. Code and benchmark are available at https://github.com/ControlNet/JRDB-AVR.

1 Introduction

JRDB-AVR addresses embodied visual reasoning where needed evidence may lie at another time, viewpoint, or person, while existing evaluations often use passive observations and score answers alone. It evaluates active observation and visual evidence, and introduces a reference agent; current baselines show answer and evidence performance can diverge.

1 Introduction

  • Limited camera views can distribute needed evidence across time, direction, or visually similar people, making answer correctness alone an unsafe signal.Plausible answers may come from priors or partial context even when the agent did not inspect the relevant object, time, or view.Source: §1 · pp. 1–2
  • Earlier benchmarks largely assess passive perception and final answers, leaving active decisions about where and when to look and evidence support under-scored.This setup can overstate embodied reasoning ability in crowded scenes with temporal ambiguity and viewpoint selection.Source: §1 · p. 2
  • JRDB-AVR requires agents to request observations across time and viewing angle, then evaluates both the final answer and grounded visual evidence.The benchmark is derived from JRDB and covers temporal search, viewpoint selection, and compositional multi-object reasoning.Source: §1 · pp. 2–3
  • JRDB-AVR-Agent plans with a structured graph, records observation-grounded facts in a world model, and solves the graph to produce answer-evidence predictions.The benchmark also reports answer, evidence, and combined correctness.Source: §1 · p. 3
  • Current baselines can show materially different answer and evidence performance, so answer accuracy alone misses unsupported success.A model may answer correctly without grounding that answer in the right visual evidence.Source: §1 · pp. 2–3

2 Related Works

Prior visual reasoning benchmarks cover compositional, temporal, and evidence-grounded tasks, but typically use fixed inputs or videos whose observations are already provided. JRDB-AVR makes active observation acquisition part of the benchmark, while its generation pipeline derives candidate questions from temporal scene graphs.

2 Related Works

  • Existing benchmarks generally test reasoning on fixed visual inputs or grounding within provided videos, whereas JRDB-AVR requires active observation acquisition and evaluates evidence support.The comparison spans temporal and viewpoint reasoning, data and camera setting, active selection, localized evidence, and evaluation target.Source: §2 · p. 3
  • JRDB-AVR question generation queries temporal scene graphs for active reasoning patterns and retains candidates after evidence, necessity, uniqueness, and manual quality checks.The graphs are built from JRDB panoramic videos and annotations, including object boxes, tracks, poses, relations, social groups, and actions.Source: Figure 2 · p. 4
  • Visual reasoning methods have progressed from monolithic VLMs to compositional programs, tool-using agents, and structured world models, but are usually evaluated on passive inputs.JRDB-AVR-Agent instantiates active evaluation with question-to-graph planning, world-model maintenance, and graph solving.Source: §2 · p. 4

3 JRDB-AVR Benchmark

JRDB-AVR evaluates whether an agent can actively acquire observations and return both an answer and supporting visual evidence. It derives questions from JRDB annotations, validates their evidence and reasoning requirements, and scores answers separately from their visual support.

3.1 Dataset Generation

  • Questions are generated by searching scene graphs built from JRDB annotations, with seven VQA generators covering temporal segments, multi-hop relations, later states, and unique-anchor searches.The source annotations include tracked objects, relations, attributes, timestamps, and viewpoint-specific evidence.Source: §3.1 · pp. 4–5, Table 2 · p. 5
  • Candidates are retained only when answers can be rederived from trusted annotations and evidence is objectively recoverable, with checks for temporal, view, multi-step, anchor, and hop necessity.The authors also manually inspect a random subset of generated questions.Source: §3.1 · pp. 5–6

3.2 Dataset Statistics

  • 2,098 active VQA questions are included in the release, generated from 27 JRDB test sequences; the engine can derive additional questions from the same source data.These counts describe the current release, not an upper bound on valid questions derivable from the annotations.Source: §3.2 · p. 6

3.3 Benchmark Protocol

  • Agents receive a question and may request observations before returning an answer and target evidence localized by timestamp and bounding box.The interface returns a 480×480 crop selected by frame and angle rather than providing the full panorama initially.Source: §3.3 · p. 6
  • Answer, evidence, and combined scores are reported separately; evidence checks target identity and, when boxes are available, bounding-box IoU, while combined correctness requires both answer and evidence to be correct.The current benchmark constructs each question around a uniquely grounded target entity.Source: §3.3 · p. 6

4 JRDB-AVR-Agent

JRDB-AVR-Agent is a training-free active reasoning baseline that builds an explicit graph plan and observation-grounded world model. It gathers and records evidence through tool interactions, then solves the plan against the accumulated graph to produce an answer and supporting evidence.

4 JRDB-AVR-Agent

  • The agent maintains an explicit graph-based world model to accumulate observed entities, attributes, relations, viewpoints, and unresolved evidence across interaction steps.The graph plan guides future observations and supports the final answer-evidence prediction.Source: §4 · p. 6
  • A schema-validated graph plan specifies the root and target nodes, candidate nodes, directed edges, and answer format, organizing observation and memory updates around an explicit reasoning target.The root defines the evidence anchor, the target defines what to recover, and edges encode the relational or temporal path.Source: §4 · p. 7
  • The world model stores entities, attributes, relations, and provenance linking graph facts to the frame and viewpoint where they were observed.It is a readable, writable, auditable representation, not a predictive future-frame or video-generation model.Source: §4 · p. 7
  • At each step, the agent can request an observation, read or write graph facts, or submit a final answer; the loop stops at a final-answer call or the tool-call budget.Graph writes are accepted only when grounded in an already observed frame-view pair; otherwise the graph state is unchanged.Source: §4 · p. 7
  • The graph solver searches accumulated observation-derived facts for a target satisfying the plan, and the proposed answer is legal only when supported by the current graph state.The final output contains both the predicted answer and supporting evidence.Source: §4 · p. 7

5 Experiments

The experiments evaluate JRDB-AVR baselines and the reference agent using answer, evidence, and combined correctness. Results show that the agent leads on all three metrics, while evidence grounding remains challenging.

5 Experiments

  • The evaluation uses 2,098 questions from 27 test sequences and compares passive, reasoning, localization-first, and tool-using baselines against an active graph-based agent.Baselines share six VLM backbones, the observation interface when applicable, and answer-evidence output schema; the reference agent uses Qwen3.5 9B.Source: §5 · pp. 7–8, Figure 3 · p. 8

5.1 Quantitative Comparison

  • 38.51 answer, 27.50 evidence, and 14.54 combined score make JRDB-AVR-Agent best across all three metrics, with gains of 4.86, 14.49, and 8.58 points over metric-specific strongest baselines.The result is reported as percentages.Source: §5.1 · p. 8, Table 3 · p. 9
  • 33.65 is the strongest baseline answer score, versus 13.01 for evidence and 5.96 for combined correctness, exposing a substantial answer-evidence gap.JRDB-AVR-Agent raises evidence to 27.50 and combined correctness to 14.54, but its answer score still exceeds its evidence score.Source: §5.1 · p. 9
  • Different ReAct backbones lead each baseline metric: Qwen3-VL 8B for answer (33.65), Qwen3-VL 4B for evidence (13.01), and Qwen3.5 9B for combined score (5.96).The lowest baseline hallucination rate is 81.22 for Qwen3.5 9B with Search-Recognize-Pipeline; the benchmark is method-agnostic.Source: §5.1 · p. 9
  • The ablation reports that both the explicit world model and active observation improve evidence-grounded reasoning and reduce conditional hallucination.The full agent uses both components; the non-active variant uses predefined passive observations, while the no-world-model variant removes persistent graph planning and the world model.Source: §5.1 · pp. 9–10, Table 4 · p. 10
  • Even with an explicit world model, the remaining errors require precise evidence binding across people, frames, viewpoints, and temporal segments.The benchmark remains challenging because questions require both selecting the right observations and grounding answers in the correct visual support.Source: §5.1 · p. 10

6 Conclusion

JRDB-AVR evaluates active observation and visual grounding through answer, evidence, combined correctness, and conditional hallucination; JRDB-AVR-Agent improves evidence grounding over VLM baselines. The benchmark evaluates recorded observations rather than live robot control.

6 Conclusion

  • JRDB-AVR reports answer, evidence, combined correctness, and conditional hallucination to expose unsupported answers that answer-only evaluation would hide.Source: §6 · p. 10
  • JRDB-AVR-Agent improves evidence grounding over VLM baselines, while the remaining gap highlights the difficulty of active observation and evidence grounding.Source: §6 · p. 10
  • JRDB-AVR evaluates active reasoning over recorded observations rather than live robot control.Source: §6 · p. 10

A Question Candidate Generator Details

The candidate generator derives questions from per-sequence scene graphs, validates them against JRDB annotations, and uses distinct templates for action boundaries, relational chains, hybrid search, unique anchors, and long-range dependencies.

A Question Candidate Generator Details

Action boundary

Chain families

Chain hybrid

  • Chain hybrid first identifies a person through a 2-hop path, then searches for the first qualifying action frame and the viewpoint that best exposes the target.It is the only generator family using frame-viewpoint pair answers, and its questions are viewpoint-sensitive.Source: A Question Candidate Generator Details · p. 17

Unique anchor and long range

  • Unique-anchor questions track the same person's later action change, presence, or distance, while long-range questions target temporally distant dependencies and numeric gap prediction.Source: A Question Candidate Generator Details · p. 17

B Additional Results

JRDB-AVR-Agent's scores vary across generator families: action-boundary questions have the strongest evidence and combined scores, while relational, hybrid, and long-range families show weaker grounding.

Generator Type Analysis

C Baseline Details

The baselines compare passive fixed-context methods with interface-using agents on the same benchmark questions, and route answers and target-grounding evidence according to each question's answer mode.

C Baseline Details

  • Four baselines and JRDB-AVR-Agent are evaluated on the same 2,098 questions; the baselines use six reported VLM backbones.Source: C Baseline Details · p. 18
  • Monolithic VLM and Chain-of-Thought receive sampled panorama frames, while Search-Recognize-Pipeline makes one proposal-driven observation and ReAct uses iterative tool use.Source: C Baseline Details · p. 18
  • Predictions pair a mode-specific answer payload with target-grounding evidence, which is scored separately from free-form explanation text.Answer payloads include choice indices, frame IDs, temporal spans, frame-viewpoint pairs, or numeric strings.Source: C Baseline Details · p. 18

C.1 Monolithic VLM

Monolithic VLM answers each question in a single generation step using sampled panorama frames as fixed visual context. It returns the routed answer and one target-person evidence box together.

  • The baseline uses sampled panorama frames as its only visual context and performs one fixed-input pass, without active observation or an intermediate reasoning stage.Source: §C.1 · p. 18
  • For each question, one structured generation step emits the routed answer and exactly one evidence box for the target person in the same JSON object.Source: §C.1 · pp. 18–19
  • Non-multiple-choice questions replace the answer field with an answer-mode-specific schema, such as frame_id or start_frame and end_frame.Source: §C.1 · p. 19

C.2 Chain-of-Thought

The Chain-of-Thought baseline adds a structured reasoning stage while retaining sampled panorama context. Search-Recognize-Pipeline instead makes one observation between a coarse panorama proposal and crop-based refinement before answering.

C.2 Chain-of-Thought

  • Chain-of-Thought reuses Monolithic VLM’s sampled panorama frames, adds a short structured reasoning stage, then conditions the answer-and-evidence stage on that summary.Source: §C.2 · pp. 19–20

C.3 Search-Recognize-Pipeline

  • Search-Recognize-Pipeline orders its stages as a coarse panorama proposal, one observation, crop-based grounding refinement, and final answer.Source: §C.3 · p. 20
  • The crop-refinement stage uses only the returned crop observation to refine the target-person proposal, without assuming a hidden gold target box.Source: §C.3 · p. 20
  • Its final answer stage receives both the selected panorama frame and the zoomed crop used to refine the proposal.Source: §C.3 · p. 21

C.4 ReAct

ReAct starts from the question without pre-sampled visual observations and alternates model actions with public observation calls. It ends with an answer and an evidence box on the latest observation.

  • ReAct repeatedly alternates between tool-use decisions and observation calls, unlike fixed-context baselines and the one-shot Search-Recognize-Pipeline workflow.Source: §C.4 · p. 21
  • When answering, ReAct emits the final answer with one evidence box relative to the most recent observation image; if none exists, it must observe first.Source: §C.4 · p. 21
  • At each iteration, the agent chooses one observe or answer action using the observation and step histories, budget state, legal frame IDs, and question.Source: §C.4 · p. 21

Related papers

Loading 2609.35032v2…