Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodology

Haochen Wang; Xiangtai Li; Zilong Huang; Anran Wang; Jiacong Wang; Tao Zhang; Jiani Zheng; Sule Bai; Zijian Kang; Jiashi Feng; Zhuochen Wang; Zhaoxiang Zhang

Source-linked AI summary

Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodology

Haochen Wang, Xiangtai Li, Zilong Huang, Anran Wang, Jiacong Wang, Tao Zhang, Jiani Zheng, Sule Bai, Zijian Kang, Jiashi Feng, Zhuochen Wang, Zhaoxiang Zhang

arXiv:2507.07999v2Jul 10, 2025cs.CVcs.AIcs.CL

TL;DR

Visual grounded reasoning lacks a benchmark that jointly tests subtle perception, traceable evidence, and second-order reasoning. The paper introduces TreeBench for this evaluation gap and TreeVGR for joint localization-and-reasoning training, reporting improvements across three benchmarks while noting scope limits in model scale and benchmark size.

Problem
Existing multimodal benchmarks do not comprehensively evaluate focused visual perception, traceable evidence, and second-order reasoning together.
Method
The paper constructs TreeBench with expert annotations and bounding boxes, then trains TreeVGR with reinforcement learning that explicitly supervises localization.
Results
TreeVGR improves V* Bench by +16.8, MME-RealWorld-Lite by +12.6, and TreeBench by +13.4 from Qwen2.5-VL-7B initialization.
Takeaways & Limitations
TreeBench provides a rigorous evaluation of traceable visual grounded reasoning, while TreeVGR enables explainable reasoning pathways through evidence supervision.
Takeaways & Limitations
TreeVGR is based on a 7B model and TreeBench contains only 405 question-answer pairs, limiting current scale and breadth.

Abstract

from arXiv · show

Models like OpenAI-o3 pioneer visual grounded reasoning by dynamically referencing visual regions, just like human "thinking with images". However, no benchmark exists to evaluate these capabilities holistically. To bridge this gap, we propose TreeBench (Traceable Evidence Evaluation Benchmark), a diagnostic benchmark built on three principles: (1) focused visual perception of subtle targets in complex scenes, (2) traceable evidence via bounding box evaluation, and (3) second-order reasoning to test object interactions and spatial hierarchies beyond simple object localization. Prioritizing images with dense objects, we initially sample 1K high-quality images from SA-1B, and incorporate eight LMM experts to manually annotate questions, candidate options, and answers for each image. After three stages of quality control, TreeBench consists of 405 challenging visual question-answering pairs, even the most advanced models struggle with this benchmark, where none of them reach 60% accuracy, e.g., OpenAI-o3 scores only 54.87. Furthermore, we introduce TreeVGR (Traceable Evidence Enhanced Visual Grounded Reasoning), a training paradigm to supervise localization and reasoning jointly with reinforcement learning, enabling accurate localizations and explainable reasoning pathways. Initialized from Qwen2.5-VL-7B, it improves V* Bench (+16.8), MME-RealWorld (+12.6), and TreeBench (+13.4), proving traceability is key to advancing vision-grounded reasoning. The code is available at https://github.com/Haochen-Wang409/TreeVGR.

1 INTRODUCTION

The paper identifies a gap in evaluating visual grounded reasoning and proposes TreeBench and TreeVGR to assess and train traceable, vision-centered reasoning. TreeBench targets subtle perception, bounding-box evidence, and second-order reasoning, while TreeVGR jointly supervises localization and answers through reinforcement learning.

Motivation: Existing multimodal benchmarks often miss fine-grained localization, traceable reasoning chains, and vision-centric second-order reasoning.Some alternatives address high-resolution inputs or simple spatial queries but lack comprehensive traceability and complex reasoning evaluation.
TreeBench: TreeBench evaluates focused perception, traceable evidence, and second-order reasoning involving object interactions, containment, and perspective transformations.Its design moves beyond isolated object localization and primitive what/where questions.
TreeBench: 1K dense-object images from SA-1B were annotated by eight experts through a semi-automated, quality-controlled process.The images provide high-resolution real-world scenes with many small and varied objects, while experts curate questions, options, and answers.
TreeBench: 3.05% is the average image area occupied by TreeBench target instances, emphasizing extremely small objects in complex real-world scenes.This small-target setting increases the need for precise visual grounding.
Results: 42.2 is Qwen2.5-VL-72B’s score on TreeBench, despite performance above 90% on some established benchmarks.The result indicates substantial remaining difficulty on TreeBench.
TreeVGR: TreeVGR uses reinforcement learning to supervise bounding-box generation alongside visual grounded reasoning.Its dual IoU reward targets both precision and recall, and the framework improves V* Bench, MME-RealWorld-Lite, and TreeBench from a Qwen2.5-VL-7B initialization.

2 RELATED WORK

Related work includes multimodal model architectures, reasoning-oriented LMM training, and benchmarks for visual reasoning. The paper argues that existing benchmarks do not jointly evaluate focused perception, traceable evidence, and second-order reasoning.

Large Multimodal Models: LMM architectures evolved from cross-attention integration toward efficient projection of visual features into an LLM’s semantic space.Flamingo and BLIP-2 exemplify cross-attention, while LLaVA uses a two-layer MLP projection.
Reasoning LMMs: Reasoning-oriented LMM research extends reinforcement-learning approaches from text-based LLMs to multimodal mathematical and scientific tasks.These approaches primarily target complex problems involving image inputs.
Benchmarks for LMMs: Current benchmarks lack comprehensive evaluation of focused perception, traceable evidence, and second-order reasoning together.Classical benchmarks often overlook fine-grained localization and verifiable reasoning chains, while other benchmarks cover only isolated aspects.
Benchmarks for LMMs: <60% is the performance of Gemini-2.5-Pro and OpenAI-o3 on TreeBench, underscoring unresolved multimodal reasoning challenges.The benchmark measures both region quality and inference beyond localization.

3 TREEBENCH

TreeBench organizes visual grounded reasoning into perception and higher-level reasoning competencies, with questions emphasizing small targets, spatial structure, comparisons, and physical interactions. Its training-pipeline figure introduces cold-start initialization and traceable-evidence reinforcement learning as TreeVGR stages.

TreeBench: TreeBench contains 405 questions covering 10 core competencies for evaluating visual grounded reasoning.The benchmark focuses on small targets, explainable reasoning pathways, and reasoning beyond precise localization.
Perception: Perception tasks test direct visual identification and attribute recognition, primarily through accurate localization of objects, regions, or text.Attributes and material tasks require fine-detail discrimination and interpretation of visual properties.
Reasoning: Reasoning tasks require aggregated visual evidence beyond recognition and localization.They include perspective transformation and ordering, among other higher-level operations.
TreeVGR: TreeVGR’s pipeline combines cold-start initialization with reinforcement learning using traceable evidence.The two stages are presented as the framework’s training structure.
Reasoning: Contact and occlusion tasks analyze physical interactions, overlap layers, and object-identification ambiguities.They rely on cues such as alignment, boundary fusion, partial coverage, and contextual constraints.
Reasoning: Spatial containment tasks evaluate hierarchical membership, surface attachment, and regional boundaries.They require interpreting object boundaries, spatial context, and containment rules.
Reasoning: Comparison tasks assess attribute differences and contextual distance across multiple co-present objects.The evaluated attributes include distance, size, and color.

4 TREEVGR

TreeVGR trains visual grounded reasoning through a two-stage pipeline that combines cold initialization with reinforcement learning and traceable bounding-box supervision. Its dual IoU reward jointly encourages recall and precision, while avoiding the need to enumerate exhaustive box predictions.

Training Pipeline: TreeVGR uses cold initialization followed by reinforcement learning to train models to localize regions of interest before answering questions.The pipeline is designed to address the inefficiency of direct reinforcement learning for visual grounded reasoning.
Training Pipeline: 32 H100 GPUs running for 48 hours were required by DeepEyes-7B for reinforcement-learning training, motivating a more computationally efficient alternative.That setup used 47K samples across 32 episodes and created barriers to broader accessibility.
Traceable Evidence: TreeVGR explicitly evaluates generated boxes against human-annotated visual evidence, guiding policies toward spatially accurate and logically coherent reasoning pathways.The IoU reward provides an interpretable measure of alignment between predicted and ground-truth regions.
Reward Design: TreeVGR’s total reward combines accuracy, formatting, and dual IoU rewards for final-answer correctness, output structure, and bounding-box quality.Formatting requires reasoning and answer spans to use the prescribed tags, while IoU compares predictions with human-annotated regions.
Reward Design: The dual IoU reward averages recall and precision terms, matching every ground-truth box to a prediction while requiring each predicted box to match at least one ground truth.This design addresses the tendency to enumerate boxes for recall and removes the need for exhaustive bounding-box enumeration.

5 EXPERIMENTS

Experiments compare TreeVGR with private, open-source, and visual grounded reasoning models across TreeBench and high-resolution benchmarks. TreeVGR performs competitively on TreeBench and improves over its Qwen2.5-VL-7B base model on V* Bench, HR-Bench, and MME-RealWorld-Lite.

Baselines: TreeBench experiments include private models, general open-source models, and recent visual grounded reasoning models as baselines.The baseline set includes GPT-4o-1120, o3-0416, Gemini models, LLaVA-OneVision, Qwen2.5-VL, InternVL3, and visual grounded reasoning systems.
TreeBench Results: TreeVGR-7B achieves performance comparable with InternVL3-78B on TreeBench.The comparison is reported among selected model results in Table 2.
High-Resolution Benchmark Results: TreeVGR achieves open-source state-of-the-art on V* Bench and significantly improves over Qwen2.5-VL-7B on HR-Bench and MME-RealWorld-Lite.The V* Bench and HR-Bench comparison appears in Table 3, while MME-RealWorld-Lite results appear in Table 4.
Ablation Studies: The ablation study evaluates the traceable training pipeline by testing the incorporation of the IoU reward into conventional reinforcement learning.This isolates the contribution of traceable evidence supervision within TreeVGR.

6 CONCLUSION

The paper introduces TreeBench for evaluating visual grounded reasoning and TreeVGR for training models with traceable evidence supervision. TreeVGR uses reinforcement learning with dual IoU rewards, while the paper notes scalability and benchmark-size limitations.

TreeBench evaluates visual grounded reasoning through focused perception, traceable evidence, and vision-centric second-order reasoning.It contains 405 high-difficulty visual question-answer pairs with precise bounding boxes and emphasizes small objects in real-world scenes.
TreeVGR advances visual grounded reasoning training by using reinforcement learning with dual IoU rewards to supervise bounding-box generation.The rewards explicitly target both localization precision and recall.
TreeVGR enables explainable reasoning pathways and achieves significant improvements across benchmarks.
TreeVGR is currently based on a 7B-parameter model, while TreeBench contains only 405 curated question-answer pairs.The paper identifies larger models and broader benchmark coverage as future directions.

ETHICS STATEMENT

The study uses publicly available, established computer-vision datasets responsibly, specifically conducting benchmarking on SA-1B under its provided licenses and intended academic purpose.

The study benchmarks on the publicly available SA-1B dataset in accordance with its licenses and intended academic purpose.

REPRODUCIBILITY STATEMENT

The paper provides implementation details and plans to release source code, datasets, and trained checkpoints to support reproducibility. Its appendices document annotation, benchmark statistics, analysis, training, and additional TreeVGR experiments.

The paper provides comprehensive model and experiment details, including training procedures and hyperparameters, in Appendix E.
The appendices document the annotation pipeline, benchmark statistics, benchmark analysis, and TreeVGR experiments.Appendix B covers three rounds of quality control, Appendix C statistics, Appendix D analysis, and Appendix F additional experiments.
TreeBench begins with 1K densely populated SA-1B images, with 100 images initially allocated per category.
After first-round manual image review, 647 images meet the category-specific relevance and quality criteria.
Two advanced LMMs generate three questions per image, producing six model-generated questions for each image before human selection.
Difficulty filtering removes questions answered correctly by all four evaluated state-of-the-art vision-language models.
Independent human cross-validation yields a final dataset of 405 rigorously validated questions.

C BENCHMARK STATISTICS

TreeBench emphasizes advanced reasoning and predominantly tests one or two tiny target instances. Its answer distribution is concentrated in four categories, while a small remainder covers categories E and F.

63% of TreeBench subtasks are advanced reasoning tasks, compared with 37% perception-related tasks.The benchmark contains 256 reasoning questions and 149 perception-related questions.
Categories A, B, C, and D account for 98.2% of answers, with A at 28.6%, B at 27.9%, C at 27.9%, and D at 13.8%.The remaining 1.8% comprises categories E and F.
44.9% of questions require identifying two target instances, while 41.5% require identifying one.Questions with three through six targets form smaller fractions of the benchmark.
The mean target-instance area is 0.0305, with most targets concentrated near zero to 0.05 of the image area.The distribution has a sharp peak near zero and a long tail extending to 0.7.

D ANALYSIS OF TREEBENCH

TreeBench evaluates fine-grained localization, traceable evidence, and second-order visual reasoning, while analyses connect localization quality to performance and test evidence dependence across benchmarks.

Benchmark characteristics: 3.05% is the average image area occupied by TreeBench target instances, confirming its focus on extremely small objects in complex scenes.The target-area histogram reports a low mean target instance area of 0.0305.
Localization and performance: Increasing mIoU from DeepEyes-7B through Pixel-Reasoner-7B to TreeVGR-7B coincides with progressively stronger overall performance.TreeVGR-7B achieves the highest mIoU and strongest overall performance simultaneously.
Localization and performance: Wrong predictions generally have smaller IoU values, while precise localization aligns closely with perception performance more than with reasoning performance.The per-question IoU analysis distinguishes overall, perception, and reasoning relationships.
Cross-benchmark analysis: TreeBench performance characteristics decouple from AI2D, MathVista, and MMStar, indicating that conventional multimodal benchmark performance does not uniformly predict TreeBench performance.The paper compares performance across all three benchmarks and reports decoupled characteristics.
Evidence quality: Masking target instances causes significant performance drops across all models, while explicit ground-truth bounding-box hints consistently improve performance.These complementary experiments support the quality and practical importance of TreeBench’s annotated visual evidence.
Results on other multimodal benchmarks: TreeVGR-7B improves substantially across most conventional multimodal benchmarks and reaches 75.3 on MMVP, surpassing Qwen2.5-VL-72B.The evaluation covers vision-centric, general VQA, document, and chart-comprehension benchmarks.

F.2 ABLATION STUDIES

The ablations show that cold initialization and dual IoU rewards contribute to visual grounded reasoning, with precision controlling repetition and recall supporting complete localization.

Cold-start initialization: Cold-start initialization benefits visual grounded reasoning on V* Bench and MME-RealWorld-Lite, including out-of-domain samples relative to the SFT dataset.The output format for target-instance bounding boxes is useful on these conventional benchmarks.
Dual IoU reward: Dual IoU rewards improve both TreeBench accuracy and mIoU from the same cold-start checkpoint, especially on out-of-distribution scenarios.The comparison is against untraceable visual grounded reasoning variants.
Precision reward: Without precision, mean response length grows rapidly because the model enumerates candidate boxes for recall and fails to produce final answers.The precision term alleviates this repetition problem.
Recall reward: Without recall, TreeBench accuracy improves but mIoU remains limited because target instances are localized incompletely.The recall term supports precise and complete localization.
Comparison with text-only RL: Traceable visual grounded reasoning outperforms vanilla text-only RL, combining pre-answer contextual grounding with accurate spatial localization.The paper identifies both factors as important for refining multimodal decision-making precision.

G LIMITATIONS AND FUTURE WORKS

The paper limits TreeVGR’s conclusions to a Qwen2.5-VL-7B architecture and identifies persistent TreeBench weaknesses in perspective-aware and fine-grained perception tasks.

Model scope: TreeVGR is evaluated with a single Qwen2.5-VL-7B model scale and architecture, leaving larger and alternative base models for future work.The authors also state that TreeVGR is not designed for ultra-long reasoning in mathematics, science, or coding.
TreeBench limitations: Perspective transform is a major TreeBench bottleneck because current models do not effectively model ego-view 3D awareness.This limitation applies even to visual grounded reasoning models.
TreeBench limitations: Attributes and material receive relatively low scores, reflecting limited basic perception in complex scenes.The paper frames perceiving details in dense visual worlds as a critical challenge.