PaperlayerView exact revision on arXiv

Source-linked AI summary

InSight-o3: Empowering Multimodal Foundation Models with Generalized Visual Search

Kaican Li, Lewei Yao, Jiannan Wu, Tiezheng Yu, Jierun Chen, Haoli Bai, Lu Hou, Lanqing Hong, Wei Zhang, Nevin L. Zhang

arXiv:2512.18745v1cs.CVcs.CLcs.LG

TL;DR

Open multimodal agents need stronger reasoning alongside perception for tasks requiring attention to scattered visual details. InSight-o3 combines a reasoning agent with an RL-trained generalized visual-search agent, improving frontier-model performance across benchmarks, including GPT-5-mini on O3-BENCH from 39.0% to 61.5%.

  • Problem

    Open multimodal agents largely focus on perception and remain limited on image tasks requiring deeper reasoning over visual details scattered across an image.

    Source: §1 · p. 1

  • Method

    InSight-o3 pairs a high-level vReasoner with an RL-trained vSearcher that locates regions described in free-form language.

    Source: §1 · p. 2, §4.1 · p. 6

  • Results

    InSight-o3-vS improves frontier-model performance across benchmarks, including GPT-5-mini on O3-BENCH from 39.0% to 61.5%.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    The specialized vSearcher can serve as a plug-and-play component for existing multimodal foundation models.

    Source: §6 · p. 10

    The average tool-call count often stays close to one, partly because vReasoner descriptions can specify regions that are easy for vSearcher to locate.

    Source: §5.2 · p. 10

Abstract

from arXiv · show

The ability for AI agents to "think with images" requires a sophisticated blend of reasoning and perception. However, current open multimodal agents still largely fall short on the reasoning aspect crucial for real-world tasks like analyzing documents with dense charts/diagrams and navigating maps. To address this gap, we introduce O3-Bench, a new benchmark designed to evaluate multimodal reasoning with interleaved attention to visual details. O3-Bench features challenging problems that require agents to piece together subtle visual information from distinct image areas through multi-step reasoning. The problems are highly challenging even for frontier systems like OpenAI o3, which only obtains 40.8% accuracy on O3-Bench. To make progress, we propose InSight-o3, a multi-agent framework consisting of a visual reasoning agent (vReasoner) and a visual search agent (vSearcher) for which we introduce the task of generalized visual search -- locating relational, fuzzy, or conceptual regions described in free-form language, beyond just simple objects or figures in natural images. We then present a multimodal LLM purpose-trained for this task via reinforcement learning. As a plug-and-play agent, our vSearcher empowers frontier multimodal models (as vReasoners), significantly improving their performance on a wide range of benchmarks. This marks a concrete step towards powerful o3-like open systems. Our code and dataset can be found at https://github.com/m-Just/InSight-o3 .

1 INTRODUCTION

InSight-o3 targets image tasks that require reasoning over scattered visual details, introducing O3-BENCH and a two-agent approach to visual reasoning and search. Its plug-and-play search model improves frontier-model performance across benchmarks.

1 INTRODUCTION

  • O3-BENCH tests reasoning-oriented tasks such as map navigation and cross-chart analysis that require gathering detailed evidence from multiple image areas while reasoning.The paper describes the benchmark as significantly harder than MME-RealWorld.Source: §1 · pp. 1–2
  • InSight-o3 divides image thinking between vReasoner for high-level reasoning and vSearcher for locating regions, including relational, fuzzy, or conceptual regions described in free-form language.Figure 1 presents a multi-step visual reasoning example; its internal reasoning is omitted.Source: §1 · p. 2, Figure 1 · p. 2
  • InSight-o3-vS improves frontier-model performance across benchmarks, including GPT-5-mini on O3-BENCH from 39.0% to 61.5%.The contributions also report that the framework surpasses OpenAI o3 across a variety of benchmarks.Source: §1 · p. 3

2 RELATED WORK

Prior multimodal benchmarks often isolate reasoning on visually simple images or perception on text-rich scenes with limited single-region lookup. O3-BENCH instead combines visual search with higher-level reasoning on dense charts and maps.

Multimodal benchmarks

  • O3-BENCH combines search and localization with higher-level reasoning on dense charts and maps, requiring cross-region evidence aggregation through interleaved, multi-hop reasoning.Earlier benchmark groups emphasize either multi-step reasoning with visually simple images or fine-grained perception with often single-region lookups.Source: §2 · p. 3

Multimodal reasoning models

  • Most multimodal reasoners remain text-centric despite recent RL-driven reasoning advances and vision-tool approaches.The related work describes cognition-centric STEM benchmarks and multimodal models using GRPO-style training, as well as tools such as Python execution.Source: §2 · p. 3

Visual search models

  • Visual-search systems have expanded from scripted detection to end-to-end and multi-turn approaches, but mostly target one region in natural images.InSight-o3 broadens this scope by decoupling visual search from reasoning and enabling multi-region search on arbitrary images.Source: §2 · p. 3

3 O3-BENCH

O3-BENCH jointly evaluates high-resolution visual perception and multi-hop reasoning using composite charts and digital maps. Its construction combines curated images with automated pre-annotation, human review, and difficulty filtering.

3 O3-BENCH

  • O3-BENCH combines high-resolution, information-dense images with multi-hop questions that retrieve and compose evidence from multiple regions.Its domains are composite charts requiring cross-chart retrieval and calculations, and digital maps requiring visual search and spatial reasoning.Source: §3 · p. 4
  • O3-BENCH comprises 204 images (117 charts, 87 maps) and 345 QA samples (163 charts, 182 maps); questions have six choices and one correct answer, with option F, No Right Choice, if none is correct.The 87 high-density map images are venue-level maps requiring legend or index reading and visual location of entities; country-, state-, and city-scale cartography is excluded.Source: §3 · p. 4, §3.1 · p. 4

3.1 SOURCE DATA COLLECTION

  • Chart images are filtered for at least eight detected layouts, while 87 high-density venue-level maps are manually collected across categories such as bus routes, campuses, and parks.The chart screening retained 256 of 2,539 MME-RealWorld candidates with sufficient sub-figures and recognizable text.Source: §3.1 · p. 4

3.2 ANNOTATION PIPELINE

  • The annotation pipeline combines automated layout and text extraction with machine-generated questions, human verification or rewriting, and difficulty filtering against three proprietary MLLMs.Reviewers check answer uniqueness, explanation fidelity, and target-layout relevance; items solved by all three models are discarded.Source: §3.2 · pp. 4–5

4 INSIGHT-O3

InSight-o3 separates high-level reasoning from visual search, training vSearcher to work with a given vReasoner. Its hybrid training combines dynamically generated search tasks with pre-generated localization data.

4 INSIGHT-O3

  • InSight-o3 assigns abstract reasoning to vReasoner and detailed region localization to vSearcher, which returns evidence for subsequent reasoning rounds.The paper focuses on training vSearcher with a strong external vReasoner to simplify optimization and improve data efficiency; it also proposes using the approach to synthesize coordinated multi-turn SFT traces.Source: §4 · pp. 5–6

4.1 TRAINING ALGORITHM

  • The hybrid RL algorithm trains vSearcher on pre-generated descriptions with ground-truth boxes out of loop and on vReasoner-generated search tasks in loop.The out-of-loop component uses IoU supervision, while in-loop tasks better match those encountered at inference.Source: §4.1 · p. 6, Figure 2 · p. 6
  • Out-of-loop rewards combine format and IoU rewards and encourage a crop-tool call; in-loop pseudo-IoU reward requires both helpful vReasoner feedback and a correct final answer.Predicted boxes below the IoU threshold receive no reward.Source: §4.1 · pp. 6–7
  • The objective follows GRPO, while in-loop advantages use global reward normalization because dynamically generated tasks lack groups.Out-of-loop advantages use group normalization; tool-response tokens are included in the importance-ratio context but masked from the loss.Source: §4.1 · p. 7

4.2 TRAINING DATA CONSTRUCTION

  • Training data pair 15,303 hard collage problems for in-loop RL with 10,186 region-description and bounding-box examples for out-of-loop RL.The collages use target images and distractors; the out-of-loop examples come from InfographicVQA layouts.Source: §4.2 · p. 7

5 EXPERIMENT

Experiments train Qwen2.5-VL-7B-Instruct as vSearcher under GPT-5-mini and evaluate it with multiple reasoning models across natural-image, mixed, and O3-BENCH tasks. Results, resolution tests, and RL ablations examine performance gains and their trade-offs.

5.1 MAIN RESULTS

  • GPT-5-mini improves 20.9% relatively on average with InSight-o3-vS, rising from 39.0% to 61.5% on O3-BENCH.Table 1 compares frontier models and systems; the reported average improvement is relative.Source: Table 1 · p. 8, §5.1 · p. 8
  • InSight-o3-vS transfers to GPT-5-nano (21.7% to 31.4% on VisualProbe-Hard; 26.5% to 34.6% on O3-BENCH; 44.3% to 51.6% overall) and retains about 7–10% lead under Gemini-2.5-Flash.The report also notes that the trained searcher generalizes when evaluated with a different vReasoner model family.Source: Table 1 · p. 8, §5.1 · p. 9
  • At 0.8M pixels (25% of training resolution), vSearcher maintains decent performance; lower resolution increases calls, while higher vReasoner resolution reduces vSearcher’s added gain.Training at one resolution and evaluating at another seems to have little impact on performance.Source: Table 4 · p. 9, Table 2 · p. 9, §5.1 · p. 9

5.2 ABLATION STUDY

  • The proposed reward and advantage design beats its ablated variants by a small average margin across three benchmarks.Ablations remove the tool condition, vReasoner feedback, outcome supervision, or global normalization.Source: Table 3 · p. 9, §5.2 · p. 10

5.1 MAIN RESULTS

  • With InSight-o3-vS, GPT-5-mini reduces its O3-BENCH performance gap with Gemini-2.5-Flash from 21.4% to 8.2%.Without vSearcher, Gemini-2.5-Flash has a larger advantage on O3-BENCH than on the other benchmarks.Source: §5.1 · p. 9

5.2 ABLATION STUDY

  • Combining in-loop and out-of-loop RL performs best; removing in-loop cuts training time per step by 80%+ but worsens average results, while removing out-of-loop also hurts performance and lengthens training.The components use different training data and sources of supervision.Source: Table 5 · p. 10, §5.2 · p. 10
  • As localization IoU and vReasoner accuracy improve, call counts trace two phases: formatting compliance first, then more accurate localization with fewer calls.Reward curves are plotted under the same “w/o feedback” setting for fair comparison.Source: §5.2 · p. 10, Figure 3 · p. 10
  • Qwen2.5-VL-7B-Instruct often avoids or misuses cropping, while rough region requests can make cropping unnecessary.The paper identifies these as reasons average tool-call counts often remain close to one.Source: §5.2 · p. 10

6 CONCLUSION

O3-BENCH evaluates visual localization and multi-hop reasoning together, and InSight-o3 separates high-level reasoning from visual search. Its reinforcement-learning-trained search agent is a plug-and-play component that helps improve frontier models.

6 CONCLUSION

  • InSight-o3 decomposes image thinking into high-level reasoning by vReasoner and visual search by vSearcher.The framework trains vSearcher to cooperate with vReasoner.Source: §6 · p. 10
  • Reinforcement-learning-trained InSight-o3-vS works as a plug-and-play component and helps improve frontier multimodal models.Source: §6 · p. 10

A.1 MULTIMODAL BENCHMARKS

The related work distinguishes benchmarks and models that emphasize reasoning from those emphasizing perception, while identifying limited support for visual reasoning across regions. It also situates InSight-o3 among agent and visual-search approaches.

A.1 MULTIMODAL BENCHMARKS

  • Classical benchmarks emphasize coarse recognition, cognition-centric benchmarks use visually simple images, and perception-centric tasks can often be solved after single-region localization.O3-BENCH targets evidence aggregation across spatially distinct regions on information-dense images.Source: §A.1 · p. 19

A.2 MULTIMODAL REASONING MODELS

  • Recent multimodal reasoning models report gains on science and math tasks, but many still focus on text-centric reasoning rather than visual reasoning.Source: §A.2 · p. 19

A.3 HIERARCHICAL AGENT FRAMEWORKS AND TOOL-USING MULTIMODAL AGENTS

  • Hierarchical agent frameworks use specialization and task decomposition, while tool-using systems orchestrate vision modules or image-manipulation code.InSight-o3 targets a complementary gap with a dedicated visual-search agent for conceptually described image regions.Source: §A.3 · pp. 19–20

A.4 VISUAL SEARCH MODELS

  • Many visual-search methods prioritize a single region in natural images; this work separates search from reasoning to support multiple distinct regions in any image.Source: §A.4 · p. 20

B.1 BENCHMARK STATISTICS

O3-BENCH is characterized by dense layouts, high-resolution images, and answer options that sometimes require ruling out every provided choice. Its machine pre-annotation pipeline detects layouts, extracts regional and global information, and synthesizes multi-region questions.

B.1 BENCHMARK STATISTICS

  • O3-BENCH averages 8.7 layouts and 2.4 target layouts per sample; chart images typically have more total layouts, whereas map images require more target layouts for reasoning.Most images have side lengths in the 2K–5K range, while some reach up to ∼10K pixels on the longer side; 7.2% use option F (No Right Choice) as the correct answer.Source: §B.1 · p. 21

B.2 DETAILS OF MACHINE PRE-ANNOTATION

  • Layout pre-annotation detects bounding boxes for structured components, using detector outputs for charts and manually reviewing and supplementing map predictions.Source: §B.2 · p. 21
  • For each layout, the pipeline crops the image and uses Qwen2.5-VL-32B to generate a caption and extract OCR text, then collects a caption and OCR text for the full image.Source: §B.2 · p. 21
  • GPT-5 receives regional triplets and global context to generate five questions with six options and step-by-step explanations, without seeing the full image.Option F is reserved for “No Right Choice.”Source: §B.2 · p. 21

B.3 MORE EXPERIMENTAL RESULTS FOR O3-BENCH

Providing target-region crops improves results for both tested models on O3-BENCH. The appendix also shows how annotations expose evidence paths, compares chart-question difficulty with MME-RealWorld, and specifies full-benchmark evaluation settings.

B.3 MORE EXPERIMENTAL RESULTS FOR O3-BENCH

  • Both GPT-5-Mini and Qwen2.5-VL-7B improve when given target-layout crops alongside the full image.The comparison is reported in Table 6.Source: Table 6 · p. 21, §B.3 · p. 21

B.4 VISUALIZATION OF O3-BENCH

  • O3-BENCH visualizations pair questions and answers with highlighted target layouts and ordered explanations that let readers verify the evidence path.The examples include map and chart items, with step-wise close-ups for large images where details may be hard to see.Source: §B.4 · p. 21, Figure 6 · p. 22

B.5 COMPARISON WITH MME-REALWORLD (CHART)

  • 38.2% average GPT-5-mini accuracy on O3-BENCH (chart), versus 82.4% on MME-RealWorld (chart), illustrates the greater challenge of O3-BENCH questions.InSight-o3 averages 3.1 versus 1.1 vSearch steps; GPT-5-mini vReasoner responses average 1942.3 versus 730.0 characters.Source: §B.5 · p. 28

B.6 FULL BENCHMARK RESULTS

  • Full-benchmark evaluation resizes oversize images to 1280×1280px for OpenAI models and 3500×3500px for other systems, with a 16K-token response budget.Incomplete responses beyond the budget count as wrong.Source: §B.6 · p. 28

C.1 IN-LOOP RL DATA

The in-loop RL data pipeline filters source examples for suitable targets and reliable answers, then synthesizes dense collages that make target localization necessary. Target tiles are kept extremely small relative to the canvas, and examples solvable without vSearcher are removed.

C.1 IN-LOOP RL DATA

  • The source pool retains about 100K items after bounding-box filtering and, for the natural-image stream, deterministic checks requiring both Qwen2.5-VL-7B and GPT-5-nano to answer correctly.Chart/OCR-centric subsets are retained from Visual CoT, while natural-image subsets and V∗ form a separate stream because of lower QA reliability.Source: §C.1 · p. 28
  • Synthesis samples and grid-quantizes a canvas, selects a feasible target scale under area and short-edge constraints, then places the target using fit-then-shrink with one canvas-enlargement fallback.The target aspect ratio receives light jitter; placement retries are bounded, and failed packing restarts from a new canvas.Source: §C.1 · pp. 29–30
  • Remaining canvas regions are panelized and filled largest-first with usage-aware image sampling, approximate aspect matching, light center-cropping, and bounded scaling.If an extra fill pass leaves residual space, synthesis resamples; otherwise it finalizes the collage.Source: §C.1 · pp. 29–30
  • Collages combine one primary target with auxiliary fill images; the target occupies less than 0.0002 of canvas area, and pass@3 removes items solvable without vSearcher.Tiles are annotated with IDs that are referenced in the question to disambiguate the target.Source: §C.1 · p. 29

C.2 OUT-OF-LOOP RL DATA

Out-of-loop RL targets are created from processed layout boxes, then described with concise visual language for search. The examples show boxed regions on infographic images, displayed side-by-side rather than stitched into collages.

C.2 OUT-OF-LOOP RL DATA

  • Layout-box processing removes trivial header and footer classes, filters oversized regions, and merges nearby boxes by enclosure effort to preserve useful neighboring context.Boxes over one quarter of image area are dropped except charts, which use a 0.8 threshold; merge effort is 1 minus summed area divided by enclosing area.Source: §C.2 · p. 30
  • Merging stops at effort thresholds of 0.15 for figure and paragraph titles and 0.1 for other boxes, while merges producing oversized or extreme-aspect boxes are skipped.Charts and tables are not merged.Source: §C.2 · pp. 30, 33
  • Figure 13 examples show pre-generated layout boxes and descriptions on infographic images, displayed side-by-side rather than stitched together.The PP-DocLayout plus-L configuration uses threshold 0.01, layout NMS, and union-mode box merging.Source: §C.2 · pp. 30, 33, Figure 13 · p. 32
  • Post-merge filtering removes nested charts or tables, uninformative unmerged boxes, regions below 0.001 of image area, and boxes outside a 1:5 to 5:1 aspect range.An image may retain multiple search targets, which become separate data entries.Source: §C.2 · p. 33
  • GPT-5-nano generates short, visually grounded region descriptions from targets marked by a red box, avoiding specific contents and small text.Descriptions may include approximate location or a region title, while ordinal or nonvisual references are discouraged.Source: §C.2 · p. 33

D.1 INSIGHT-O3 IMPLEMENTATION DETAILS

The implementation constrains image sizes, tool calls, response lengths, and training settings for vSearcher. Qualitative examples show the reasoner and searcher iteratively assembling localized evidence before reaching an answer.

D.1 INSIGHT-O3 IMPLEMENTATION DETAILS

  • vSearcher training uses a 1:1 mix of in-loop and out-of-loop data for 150 fully on-policy steps, while freezing Qwen2.5-VL-7B-Instruct’s vision tower and adapter.Training uses batch size 24, 8 rollouts, learning rate 10^-6, KL coefficient 0.01, and reward weights 0.2 for format and 0.8 for IoU.Source: §D.1 · p. 33
  • Training and evaluation allow at most 6 tool calls; vSearcher image limits are about 3.2M pixels in training and 12.8M pixels in evaluation, with oversized images downsampled.Maximum response lengths are 9K and 32K tokens for training and evaluation, respectively.Source: §D.1 · p. 33

D.2 MORE VISUALIZATIONS OF INSIGHT-O3 REASONING PROCESS

  • Across a few rounds, vReasoner requests natural-language targets and vSearcher localizes evidence and returns snippets that support the final answer.The examples use GPT-5-mini as vReasoner and show venue-level cues such as legend or index lookups.Source: §D.2 · p. 33, Figure 14 · p. 34, Figure 15 · p. 35

D.3 COMPARATIVE ANALYSIS BETWEEN INSIGHT-O3 AND BASELINES

This section compares InSight-o3 with baselines and describes the evaluation materials and procedures, including O3-Bench construction, prompts and agent interaction formats, answer verification, and benchmark profiles. It also examines visual-search behaviors and failure cases.

D.3 COMPARATIVE ANALYSIS BETWEEN INSIGHT-O3 AND BASELINES

  • Without vSearcher, GPT-5-mini often hallucinates visual details; vanilla Qwen2.5-VL-7B vSearcher reduces factual errors but can return wrong crops or give up, leading to wrong answers.The comparison covers three O3-Bench examples.Source: §D.3 · p. 36
  • InSight-o3-vS reasons about requested regions, checks crops, and can correct bad crops; its usually medium-to-high-quality crops focus vReasoner and support evidence-based reasoning.In the map example, it reasons that the box should cover the legend/index and the “Catering venues” list; in some cases it adjusts a crop that misses the intended area.Source: §D.3 · p. 36

D.4 FAILURE CASES OF INSIGHT-O3

  • In the first three failure cases, vReasoner errs despite correct crops; the remaining failures involve vSearcher misunderstanding map structure or complex region descriptions, and most O3-Bench errors are attributed to vReasoner.The cited examples include ignoring visual evidence due to internal-knowledge bias and concluding without examining key information.Source: §D.4 · p. 36, Figure 20 · p. 41, Figure 21 · p. 42

D.5 INSIGHT-O3 WITH QWEN3-VL VREASONER

  • InSight-o3 outperforms both base Qwen3-VL-32B and Qwen3-VL-32B paired with Qwen2.5-VL-7B without reinforcement learning, with a relatively small O3-Bench advantage.The passage provides no numerical values for these comparisons.Source: Table 8 · p. 43, §D.5 · p. 43

E.1 PROMPTS FOR INFORMATION EXTRACTION

  • The information-extraction prompt requires image-type identification, a comprehensive factual caption, and complete OCR, with rules for fidelity, reading order, and uncertain text.It specifies dense structured captions and extraction of visible text, including tables in Markdown format.Source: §E.1 · pp. 43–44

E.2 PROMPTS FOR CONSTRUCTING O3-BENCH

  • The O3-Bench construction prompts require image-grounded multiple-choice questions that combine at least two visual reasoning hops across layouts or global and local text.The chart prompt calls for careful scanning, cross-referencing, and light calculations; the map prompt gives examples such as linking legends to routes or grid indices to labels.Source: §E.2 · pp. 45–46
  • O3-Bench construction prompts require image/OCR-grounded, diverse questions that avoid at-a-glance answers and use exact map labels, map-based scales, and a single correct option.The answer-option rules specify six choices, A–F, with F always “No right choice”; explanations should show step-by-step reasoning and cite the layouts used.Source: §E.2 · p. 47

E.3 PROMPTS FOR EVALUATION

  • GPT-5-nano’s VisualProbe-Hard accuracy drops from 21.7% to 16.0% with its default system prompt; O3-Bench uses default prompts, while other benchmark differences are mostly within 1%.For open models, default prompts are used because they are relatively prompt-sensitive, and the thinking prompt often underperforms for Qwen3-VL models.Source: §E.3 · p. 47

F.1 VREASONER PROMPTS

  • vReasoner is prompted to identify needed visual facts, request one concise region at a time when information is unclear, and critically assess returned crops before refining or answering.During training, it reports tool feedback; during inference, the prompts omit that feedback. Search failure prompts it to revise the region description or answer from available evidence.Source: §F.1 · pp. 48–50

F.2 VSEARCHER PROMPTS

  • vSearcher locates a described region, may zoom in, and returns bounding-box coordinates, using [0, 0, 0, 0] when the target cannot be found.Without RL, Qwen2.5-VL-7B has difficulty following the standard prompts, so its evaluation uses a simpler region-to-bounding-box prompt.Source: §F.2 · pp. 51–52

F.3 PROMPTS FOR ANSWER VERIFICATION

  • Answer verification uses GPT-5-nano to compare a model’s answer with the ground truth, returning <correct> for visual or semantic matches and <wrong> for mismatches or uncertainty.The prompt provides the question, ground-truth answer, and model answer, and requires only one of the two labels.Source: §F.3 · p. 52

G.1 NATURAL-IMAGE BENCHMARKS

  • V⋆-Bench uses 191 SA-1B natural images for fine-grained attribute recognition and spatial reasoning, while Tree-Bench uses 405 QA pairs with bounding-box evidence and second-order reasoning.V⋆-Bench targets high-resolution visual details; Tree-Bench emphasizes traceable evidence and object interactions or spatial hierarchies.Source: §G.1 · p. 52
  • VisualProbe-Hard contains 106 questions on images with tiny targets and distractors, often requiring iterative search across regions rather than a single glance.VisualProbe organizes questions into easy, medium, and hard subsets; the hard subset emphasizes active search such as zooming or sequential exploration.Source: §G.1 · p. 52

G.2 MIXED BENCHMARKS

  • HR-Bench tests ultra-high-resolution images through Fine-grained Single-instance Perception and Fine-grained Cross-instance Perception, with 100 queries per sub-task on 8K images and 4K versions.FSP includes object attributes, OCR, and visual prompts; FCP includes maps, charts, and spatial relationships.Source: §G.2 · p. 53
  • MME-RealWorld covers 13,366 images and 29,429 QA pairs across 43 task types and five scenarios; this evaluation uses its official lite version with 50 samples per task.The benchmark spans real-world applications including driving, surveillance, remote sensing, and sports; the lite subset is used for efficiency.Source: §G.2 · p. 53

G.3 COMPARISON OF O3-BENCH AND RELATED BENCHMARKS

O3-BENCH is harder than related benchmarks on overlapping chart and map domains, while its scope is limited to composite charts and digital maps. These domains pair structured images requiring abstract reasoning with less-structured maps requiring visual reasoning.

G.3 COMPARISON OF O3-BENCH AND RELATED BENCHMARKS

  • 4602×3967 vs 1875×1269 average image resolution, 39.0% vs 83.8% average GPT-5-mini accuracy, and 2.9× as many visual search steps distinguish O3-BENCH from MME-RealWorld.The comparison covers their overlapping chart and map domains.Source: §G.3 · p. 54
  • O3-BENCH is on par in scale with fine-grained perception benchmarks and several commonly used benchmarks across other multimodal areas.The passage lists examples including V⋆-Bench (191), Tree-Bench (405), and MathVision test-mini (304).Source: §G.3 · p. 54
  • O3-BENCH focuses on composite charts and digital maps, pairing structured images that demand abstract reasoning with less-structured maps requiring visual reasoning.The authors argue this scope is generally sufficient for evaluating digital-domain image thinking; natural-image benchmarks already exist.Source: §G.3 · p. 54

Related papers

Loading 2512.18745v1…