PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memory

Yu Zhou, Cheng-Fu Yang, Rui Sun, Di Wu, Sibo Peng, Mingyang Zhang, Zhen Yang, Ruiyi Zhang, Nanyun Peng, Zhe Gan, Kai-Wei Chang

arXiv:2610.26860v1cs.CLcs.AIcs.CVcs.LGcs.MM

TL;DR

Many visual tasks need grounded perception, external knowledge, or numerical measurements that internal reasoning over the input alone may not reliably provide. OpenVLHarness combines task-level tool interfaces and evidence rendering with persistent multimodal memory and computation over stored artifacts. Across 22 benchmarks, it improves average performance by 9.7–12.6 percentage points over base MLLMs.

  • Problem

    Many visual tasks require grounded perception, external knowledge, or numerical measurements that MLLMs cannot reliably infer from the input alone, and heterogeneous tool outputs must be interpretable and reusable.

    Source: §1 · p. 2

  • Method

    OpenVLHarness combines backend-independent task-level tool interfaces and model-facing evidence rendering with persistent multimodal memory and a coding agent that computes over stored artifacts.

    Source: §1 · p. 2

  • Results

    Across 22 benchmarks, OpenVLHarness improves average performance by 9.7–12.6 percentage points over base MLLMs and outperforms existing visual tool-use baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Component ablations show incremental benefits from both the capability layer and multimodal memory.

    Source: §5 · p. 10

Abstract

from arXiv · show

Multimodal large language models (MLLMs) have made rapid progress on visual reasoning, yet many tasks require evidence that is difficult to obtain through internal reasoning alone: our controlled study on 22 benchmarks shows that simply increasing the inference budget of the base model yields only modest gains. Such tasks demand grounded perception, external knowledge, and numerical measurements obtained through heterogeneous tools, whose outputs must be made interpretable to the model and retained across reasoning steps. We introduce OpenVLHarness, a multimodal reasoning harness organized around a capability layer and persistent multimodal memory. The capability layer combines task-level tool interfaces with evidence rendering, providing consistent access to heterogeneous backends and converting their outputs into model-readable observations. Persistent multimodal memory retains the corresponding images, structured data, and web references as addressable artifacts throughout each query. A coding interface executes generated programs over stored data, enabling computations that combine results from multiple tools. Together, these mechanisms support an iterative process of acquiring, inspecting, and composing multimodal evidence. Evaluated on 22 benchmarks with eight open and proprietary backbones, OpenVLHarness improves average scores by 9.7-12.6 points over matched no-tool baselines, including +11.2 with Qwen3-VL-8B, +9.7 with Kimi K3, and +9.9 with GPT-6 Astra. With GPT-6 Luna at medium reasoning effort, OpenVLHarness outperforms the base model, Codex, and Visual Sketchpad at maximum effort by 7.7-8.5 points while incurring at most 37% of their model-token cost.

1 INTRODUCTION

Many visual tasks need grounded perception, external knowledge, or numerical measurements that are difficult to obtain from the input alone, while increased reasoning effort yields diminishing returns. OpenVLHarness combines consistent tool interfaces and evidence rendering with persistent artifact memory, and improves average performance across 22 benchmarks and eight backbones.

1 INTRODUCTION

  • Visual tasks may require locating tiny objects, retrieving long-tail knowledge, or computing quantities from measurements, which calls for evidence acquired through perception, search, and computation.Increasing the base model's reasoning effort from xhigh to max multiplies model-token spending by 2.7 for only a 0.7-point average-score improvement.Source: §1 · p. 2
  • The capability layer combines backend-independent task interfaces with rendered observations, while persistent memory retains artifacts for reuse and computation across turns.The Python Coding Agent executes generated programs over stored data so results from multiple tools can be composed.Source: §1 · pp. 2–3
  • OpenVLHarness improves average performance by 9.7–12.6 percentage points over base MLLMs across 22 benchmarks and outperforms existing visual tool-use baselines.The evaluation uses eight open and proprietary backbones.Source: §1 · p. 2
  • Figure 2 depicts the orchestrator invoking capabilities whose outputs are rendered as observations, while memory keeps images, structured data, and web references addressable for later calls.An Environment Update advertises available artifacts for reuse, including in memory-grounded computation.Source: Figure 2 · p. 3

2 HARNESS DESIGN

OpenVLHarness connects capability calls to inspectable evidence and retains the underlying artifacts so later steps can reuse and compute over them. Its design centers on a capability layer and persistent multimodal memory.

2 HARNESS DESIGN

  • The harness pairs consistent capability interfaces and model-facing observations with persistent artifact memory; a coding agent connects outputs from different capabilities into multi-step procedures.The modular interfaces and renderers also support specialization through text edits and tool composition.Source: §2 · p. 3, §2.1 · p. 3, §2.2 · p. 4

2.1 CAPABILITY LAYER

  • Task-level interfaces define operations independently of backend, allowing alternatives such as SAM 3 or Grounding DINO to preserve the orchestrator's calling convention.The base instantiation includes nine capabilities spanning perception, information seeking, and computation.Source: §2.1 · pp. 3–4
  • Output processors turn heterogeneous tool results into observations combining visual attachments, concise descriptions, structured information, and reusable-artifact references.Grounding can return a labeled mask overlay and normalized boxes, while depth estimation can return a colorized map; underlying masks and arrays remain available for later computation.Source: §2.1 · p. 4

2.2 PERSISTENT MULTIMODAL MEMORY

  • The artifact registry retains image artifacts, structured data bindings, and web references under handles, while Environment Updates advertise available artifacts without repeating their full payloads.Later calls resolve handles into the image, URL, or data values required by the receiving tool.Source: §2.2 · p. 4
  • A Python Coding Agent runs generated programs over named memory bindings, allowing computations such as combining grounding masks with depth arrays without serializing large arrays into the orchestrator's request.Figure 3 illustrates chaining camera poses, depth, and grounding for 3D measurement; passing grounded crops through OCR and image search before web verification; and reconciling masks to correct an undercount.Source: §2.2 · p. 5, Figure 3 · p. 5

3 EXPERIMENTS

Across 22 benchmarks, OpenVLHarness improves on base models and training-free harnesses, while component studies, failure analysis, and domain-specific optimization examine how its mechanisms and adaptations contribute. Results also show that increased reasoning effort alone does not replace the harness.

3.1 TASKS, MODELS, AND BASELINES

  • The evaluation spans four task domains and 22 datasets, comparing eight open and proprietary backbones with no-tool, training-based, and training-free baselines.Qwen3-VL results are means of three runs; proprietary-backbone results are single runs because of API cost.Source: §3.1 · pp. 5–7

3.2 MAIN RESULTS

  • Across evaluated backbones, OpenVLHarness improves average performance over base models and outperforms training-free harnesses; Qwen3-VL-8B rises from 34.2 to 45.4 (+11.2).It also exceeds OctoTools (31.8) by 13.6 points on Qwen3-VL-8B; the reported GPT-6 Astra average rises from 60.3 to 70.2.Source: Table 1 · p. 6, §3.2 · p. 7

3.3 CAN MORE REASONING REPLACE THE HARNESS?

  • At medium effort, GPT-6 Luna with OpenVLHarness scores 56.5, above Base, Codex, and Visual Sketchpad at maximum effort, while using at most 37% of their model-token cost.The comparison counts model tokens, including reasoning tokens, but excludes tool execution and judging.Source: Table 2 · p. 7, §3.3 · p. 8

3.4 ABLATION STUDY

  • Adding raw tool access, then the capability layer, raises overall averages by +3.8/+4.3 and a further +3.4/+3.2 points at 8B/32B, respectively.Completing persistent multimodal memory with reusable image handles adds another +3.1/+4.5 points, bringing the full harness to 47.1/52.5.Source: §3.4 · p. 8, Table 3 · p. 8

3.5 FAILURE ANALYSIS

  • Failure patterns vary by domain: crowded-image perception errors account for 75% of failures there, while visual search and deep research most often fail through omitted tool calls (49%).Unusable or ignored correct tool outputs are uncommon (3–8% across domains).Source: §3.5 · p. 9, Figure 4 · p. 9

3.6 HARNESS OPTIMIZATION

  • Failure critiques guide two specialization workflows: harness search edits prompts and rendering templates, while tool generation adds task-specific operations; their benefits align with dataset remedy profiles.Search improves ERQA/RealWorldQA by +4.1/+3.7 points, while generated tools improve SparBench/FREAK-VQA by +8.2/+4.5.Source: §3.6 · pp. 9–10, Figure 4 · p. 9

4 RELATED WORK

Prior visual tool-use systems range from fixed program-synthesis pipelines to multi-turn frameworks and manually designed training-free harnesses. OpenVLHarness emphasizes adaptable evidence processing and tool composition, while extending referenceable working memory to multimodal artifacts.

Visual Tool-Use Baselines

  • Program-synthesis visual tool-use systems compile questions into fixed tool plans that cannot adapt to execution results.Later multi-turn frameworks interleave reasoning with visual feedback but often rely on brittle task-specific designs and extensive training.Source: §4 · p. 10
  • Visual Sketchpad and OctoTools are closest as training-free frameworks, but use manually designed tool interfaces and execution strategies.OpenVLHarness instead focuses on specializing evidence processing and tool composition as tools, models, and tasks evolve.Source: §4 · p. 10

Working Memory for Tool-Using Agents

  • OpenVLHarness extends referenceable intermediate results to a unified multimodal registry of images, structured data, and web evidence for later inspection and reuse.Earlier agent approaches use referenceable variables, executable-interpreter variables, or symbolic references to intermediate tool outputs.Source: §4 · p. 10

5 CONCLUSION

OpenVLHarness combines a capability layer with persistent multimodal memory to acquire, inspect, and compose evidence across tool calls. Its behavior analysis shows that tool engagement and capability use vary by model and domain.

5 CONCLUSION

  • The harness combines task-level tool interfaces and evidence rendering with persistent artifacts that can be reused across turns and computations over stored data.The conclusion describes broad gains across 22 benchmarks and eight backbones, component-ablation benefits, and domain specialization without updating MLLM weights.Source: §5 · p. 10

A TOOL-USE BEHAVIOR

  • The behavior analysis covers five models and approximately 123,000 rollouts with 224,000 calls, using exact counts and defining engagement as at least one call.Per-question comparisons are paired, but tool-selection rows are agent-chosen and the reported per-capability quantities are correlational, not causal.Source: A TOOL-USE BEHAVIOR · p. 19

A.1 TOOL-CALL VOLUME AND ENGAGEMENT

  • 28.9% of Qwen3-VL-8B questions and 25.3% of GPT-6 Luna questions received no call, versus 6.2%, 4.8%, and 2.6% for the larger models.Engagement varies with model scale within a family, not across families; grounded perception engages nearly every model, while smaller models leave more spatial-reasoning and general-VQA questions untouched.Source: §A.1 · p. 19, Figure 5 · p. 19

A.2 DISTRIBUTION OF TOOL CALLS

  • Visual Grounding is the largest aggregate call category for every model, accounting for 45% of GPT-6 Sol calls and 63% of Qwen3-VL-32B calls.GPT-6 Sol and Astra allocate 20% and 18% of calls to Zoom-in, while web retrieval is concentrated in visual search and deep research.Source: §A.2 · p. 20, Figure 6 · p. 20
  • Grounded perception is dominated by Visual Grounding, visual search and deep research by web capabilities, and spatial reasoning by four capabilities with comparable load.GPT-6 Sol and Astra are exceptions to Visual Grounding dominance in grounded perception, allocating roughly a quarter of that domain's calls to Zoom-in.Source: §A.2 · p. 20, Figure 7 · p. 21

A.3 EFFECT OF TOOL USE ON ACCURACY

OpenVLHarness improves paired per-question outcomes, though capability-level comparisons are correlational because the agent selects which questions receive each tool. Its call patterns often combine capabilities, and additional-backbone results include consistent gains with GPT-6 Sol.

A.3 EFFECT OF TOOL USE ON ACCURACY

  • All five models improve by +6.2 (GPT-6 Astra) to +10.1 (Qwen3-VL-8B) points; between 2,336 and 4,649 questions are corrected and 700 to 2,146 degraded per model.The calls-per-net-gain range is 11.7 calls (GPT-6 Luna) to 28.1 calls (Qwen3-VL-32B).Source: §A.3 · p. 22, Table 5 · p. 22
  • No-call paired differences are near zero, ranging from −1.8 to +1.8 points; GPT-6 Astra's +1.8 is based on 614 questions.The two larger negative values, for Qwen3-VL-32B and Qwen3-VL-8B, indicate a small cost when the tool interface is carried but unused.Source: §A.3 · p. 22
  • For every model, paired accuracy differences on questions invoking Image Search are +18.9 to +27.1 points, versus corresponding non-invoking-question controls ranging from −0.6 to +6.6 points.Source: §A.3 · p. 22

A.4 CAPABILITY COMPOSITION

  • Among engaged rollouts, two or more distinct capabilities are used in 11.6% to 31.9%; cross-family combinations occur in 4–16% and usually pair perception with Python computation.These combinations commonly ground or measure first, then compute over the resulting quantities.Source: §A.4 · p. 24

B.1 ADDITIONAL BACKBONES

  • With GPT-6 Sol, OpenVLHarness improves every one of the 22 benchmarks, raising the average from 52.7 to 64.3.The evaluation uses the same tools, memory, prompts, and scoring as the other Table 1 backbones.Source: §B.1 · p. 24, Table 7 · p. 25

B.2 ABLATION STUDY DETAILS

The ablation uses a strict superset ladder to isolate successive harness mechanisms while holding the backbone, tools, and decoding configuration fixed.

B.2 ABLATION STUDY DETAILS

  • Each ablation rung adds exactly one mechanism to the preceding variant, so adjacent gaps measure that mechanism's marginal value.The variants share the same nine-tool set, backbone weights, and decoding configuration.Source: §B.2 · p. 26
  • The ladder progresses from no tools, to raw tool access, to the capability layer with model-readable post-processing and restored schemas.The raw-tool variant has no persistence; the capability-layer variant still carries nothing across turns.Source: §B.2 · p. 26
  • Textual memory adds environment updates, URL references, and typed data bindings, while full OpenVLHarness adds addressable tool-generated image references.Textual memory replaces generated images with descriptions; image references enable compositions such as grounding, zoom, OCR, and search.Source: §B.2 · p. 26, Table 8 · p. 26
  • The evaluation covers 22 datasets, uses the same fixed-seed subsamples across variants for datasets over 500 samples, and averages three independent runs.Domain averages are macro-averages over datasets in each domain.Source: §B.2 · p. 26

B.3 TRAINING-BASED BASELINES

On the Qwen2.5-VL-7B-Instruct backbone, the training-free OpenVLHarness exceeds three fine-tuned harnesses on the 22-dataset average. The comparison also shows narrower gaps in familiar task domains and a benefit that depends on backbone strength.

B.3 TRAINING-BASED BASELINES

  • 33.45 is OpenVLHarness's 22-dataset average, ahead of PixelReasoner (28.02), AdaReasoner (25.29), and DeepEyesV2 (25.02), without weight updates.The compared training-based harnesses fine-tune the same Qwen2.5-VL-7B-Instruct backbone.Source: §B.3 · p. 27, Table 9 · p. 27
  • The margin is largest on grounded perception (23.05 vs. at most 10.41) and visual search and deep research (19.91 vs. at most 15.59), but narrows on general VQA and hallucination (50.58 vs. 49.32) and spatial reasoning (39.54 vs. 39.37).The training-based harnesses rarely produce the dense box lists required by FSCD and OdinW and do not retrieve web evidence in this setup.Source: §B.3 · p. 27
  • OpenVLHarness reaches 7.80 AP on OdinW-13-All with Qwen2.5-VL-7B, versus 50.40 with Qwen3-VL-8B, because tool planning and output conventions rely on the backbone.The harness can adopt a new backbone without retraining.Source: §B.3 · p. 28

B.4 GROUNDED PERCEPTION ACROSS OBJECT COUNTS

Common counting and grounding benchmarks concentrate on few objects, while OpenVLHarness improves counting and grounding in denser settings. Performance still drops sharply at very high counts.

B.4 GROUNDED PERCEPTION ACROSS OBJECT COUNTS

  • Widely used counting benchmarks largely contain ten or fewer objects, and common grounding benchmarks are concentrated at one target per sample or image.Only FSC-147 and FSCD-LVIS have substantial counting mass above 50 objects; dense, many-object grounding appears only in FSCD variants.Source: §B.4 · p. 28, Figure 9 · p. 28, Figure 10 · p. 29
  • At 8B, FSC-147 exact-match accuracy rises from 5.0 to 32.9 for 21–50 objects; on FSCD-LVIS, OpenVLHarness reaches F1 above 56 up to 100 instances and around 41 beyond.FSCD-LVIS base models reach F1 ≤12.5, and accuracy falls steeply with count for every configuration; above 100 objects, exact counting remains essentially unsolved.Source: §B.4 · p. 30, Table 10 · p. 30
  • On LVIS federated val, OpenVLHarness raises AP from 18.1 to 45.2 at 8B and from 22.9 to 45.5 at 32B.It also lowers FSC-147 mean absolute error and raises CountQA relative accuracy, indicating improvement beyond near-miss corrections.Source: §B.4 · p. 30, Table 11 · p. 30

B.5 EXTENDED VISUAL SEARCH AND DEEP RESEARCH RESULTS

Across 11 information-seeking benchmarks, the agent workflow improves results for every evaluated backbone. Gains are largest for open models, and Qwen3-VL-32B with the harness exceeds direct GPT-5 on average.

B.5 EXTENDED VISUAL SEARCH AND DEEP RESEARCH RESULTS

  • Across 11 benchmarks, average scores rise from 13.74 to 29.05 for Qwen3-VL-8B, 19.21 to 34.61 for Qwen3-VL-32B, and 31.58 to 41.80 for GPT-5.The benchmark set includes five information-seeking benchmarks not included in Table 1.Source: §B.5 · p. 30, Table 12 · p. 31
  • With the harness, Qwen3-VL-32B scores 34.61 on average, exceeding direct GPT-5 at 31.58.The reported gains are largest for open models, whose parametric knowledge is most limited.Source: §B.5 · p. 30

B.6 REASONING-EFFORT STUDY: SETUP AND FULL RESULTS

Across 22 benchmarks, OpenVLHarness at medium reasoning effort outperforms all other methods at every tested effort setting. The study also compares token costs, usage patterns, and benchmark-level outcomes.

B.6 REASONING-EFFORT STUDY: SETUP AND FULL RESULTS

  • The GPT-6 Luna study tests five effort settings on all 22 benchmarks, with 26,229 questions per setting and 524,580 questions total.Costs include model tokens, including hidden reasoning tokens, but exclude tool execution and judging; 134 questions that failed on every retry counted as incorrect.Source: §B.6 · p. 31, Table 16 · p. 35
  • At medium effort, OpenVLHarness scores 56.52 at $0.92 per 1,000 questions, exceeding every other method’s settings by at least 7.68 points; Visual Sketchpad at max scores 48.84 at $5.15.These are 22-benchmark averages and model-token costs at Standard-tier prices.Source: §B.6 · p. 31, Table 13 · p. 32
  • OpenVLHarness leads Base in every domain from medium effort onward and scores best on 20 of 22 benchmarks; at low effort, it trails Base in general VQA and hallucination and spatial reasoning.At max effort, its largest margins over the best other method are 15.6 points in grounded perception and 12.4 in visual search and deep research; HallusionBench and ERQA are the exceptions to its 20-benchmark lead.Source: §B.6 · p. 33
  • From low to max effort, OpenVLHarness input tokens grow from about 14,500 to 24,600 per question, while Base stays near 1,800 input tokens and raises reasoning tokens from 301 to 5,852.Codex input grows from about 8,000 to 11,800 tokens per question, and Visual Sketchpad from 20,800 to 24,100.Source: §B.6 · p. 33, Table 14 · p. 33
  • Codex executes at least one command on 0.16% of questions at low effort and 6.17% at max, and its average remains at most 2.0 points below Base from medium to xhigh effort.At low effort it trails Base by 7.4 points; at max it scores 0.5 points above Base, with every question receiving a non-empty answer.Source: §B.6 · pp. 33–34, Table 15 · p. 34
  • Reported cost comparisons cover model-token costs only, and each result-table cell is a single run.Tool-backend costs are excluded; the service tier affects price and latency, not the model.Source: Table 13 · p. 32, §B.6 · p. 34, Table 16 · p. 35

B.7 CODEX WITH OPEN-SOURCE BACKBONES

With Qwen3-VL-8B and Qwen3-VL-32B, Codex scores below Base on average, while OpenVLHarness scores higher than Codex on nearly all benchmarks. These Codex results measure an agent loop without shell-based tools.

B.7 CODEX WITH OPEN-SOURCE BACKBONES

  • In these runs, Codex’s sandbox could evaluate only simple expressions and could not start shell processes, so results measure its agent loop and prompt without shell-based tools.The setup used medium effort, disabled web search, and capped each response at 2,048 tokens.Source: §B.7 · p. 36
  • Across 22 benchmarks, Codex scores below Base with Qwen3-VL-8B (31.42 vs. 34.24) and Qwen3-VL-32B (36.46 vs. 38.30), while OpenVLHarness reaches 45.41 and 50.92.OpenVLHarness exceeds Codex on every benchmark with the 8B model and on 20 of 22 with the 32B model.Source: §B.7 · pp. 36–37, Table 17 · p. 36

C HARNESS IMPLEMENTATION DETAILS

The implementation exposes tools through stable capability interfaces, retains their outputs in reusable multimodal memory, and supports computation over prior tool results. The appendix details search, perception, coding, memory, prompting, and model-serving configurations.

C HARNESS IMPLEMENTATION DETAILS

  • Each capability accepts typed arguments and returns a natural-language message plus memory artifacts, while configurable backends can be switched without changing the agent’s interface.Experiments use the default backend for each capability.Source: C HARNESS IMPLEMENTATION DETAILS · p. 38, Table 18 · p. 38

C.1 SEARCH TOOLS

  • Search tools retrieve and summarize results, while Webpage Visit extracts goal-relevant content from fetched pages.Text Search routes queries to appropriate endpoints and supports follow-up visits to cited URLs; Image Search performs reverse-image search.Source: §C.1 · pp. 38–39

C.2 PERCEPTION TOOLS

  • Perception tools crop localized regions, ground text queries in images, extract text, estimate depth, and describe camera motion across ordered views.Camera Trajectory requires at least two images and sufficient scene overlap; its metric estimate depends on model calibration.Source: §C.2 · p. 40

C.3 PYTHON CODING AGENT

  • The Python Coding Agent executes generated code over prior tool outputs, enabling aggregation, filtering, and cross-tool numerical reasoning.By default, the orchestrator’s backbone generates code in a single-shot session.Source: §C.3 · p. 40

C.4 MULTIMODAL MEMORY ARTIFACTS

  • Persistent memory stores heavy artifacts once and lets the agent reuse them through handles that resolve to images, web references, or data.Environment Updates re-advertise the live memory namespace after tool turns.Source: §C.4 · p. 40, Table 19 · p. 41

C.5 SYSTEM PROMPT

  • The system prompt directs the agent to use a Perception-Reasoning-Action framework and make at most one native tool call per turn.It also instructs the agent to verify whether each tool result is informative and not to guess from empty or uninformative results.Source: §C.5 · p. 42

C.6 MODELS AND SERVING

  • Experiments use open-source and proprietary backbones with shared tools and memory, while search summarization uses Qwen3-VL-8B-Instruct and code generation uses the orchestrator’s backbone.The agent is capped at fifteen tool-calling iterations per query; benchmark judges vary by task, and grounding uses COCO-style box AP.Source: §C.6 · p. 43

D BENCHMARKS AND EVALUATION PROTOCOL

The evaluation spans 22 benchmarks, with common agent settings and task-specific scoring. The appendix describes grounded perception, visual search and deep research, and general VQA and hallucination benchmarks.

D BENCHMARKS AND EVALUATION PROTOCOL

  • The evaluation covers 22 benchmarks, using a shared agent configuration and consistent prompts and scorers across methods and backbones on each benchmark.Qwen3-VL results average three seeded runs; trajectories exceeding the tool-call budget or failing after retries score incorrect.Source: D BENCHMARKS AND EVALUATION PROTOCOL · p. 44, Table 21 · p. 44

D.1 GROUNDED PERCEPTION

  • Grounded perception includes dense-scene counting and detection benchmarks, scored with exact-match counting accuracy and COCO box AP.OdinW-13-All asks for every listed category in a single query, so its results are not directly comparable to one-category-per-query results.Source: §D.1 · pp. 44–45

D.2 VISUAL SEARCH AND DEEP RESEARCH

  • Visual search and deep research benchmarks test knowledge-seeking, multimodal retrieval, and multi-step research; search tools did not surface gold answers verbatim.GPT models’ scores are affected to some extent by refusals on identity queries, which the judge counts as incorrect.Source: §D.2 · pp. 46–47

D.3 GENERAL VQA AND HALLUCINATION

  • General VQA and hallucination benchmarks include everyday visual questions and tests designed to expose reliance on language priors over image evidence.HallusionBench reports an Overall score averaging question, question-group, and figure accuracies.Source: §D.3 · p. 48

D.4 SPATIAL REASONING

The appendix specifies spatial-reasoning evaluation and describes a failure-analysis protocol for specializing the harness through text changes or new tools.

D.4 SPATIAL REASONING

  • Spatial reasoning uses multiple-choice benchmarks, including multi-view tasks, and follows the EASI leaderboard split and implementation.SparBench averages 20 per-task metrics equally across its task types; MindCube uses the 1,040-question MindCube-Tiny split.Source: §D.4 · pp. 48–49

E FAILURE ANALYSIS AND HARNESS OPTIMIZATION DETAILS

  • The failure-analysis appendix describes critique-guided specialization through edits to harness text or additions of task-specific tools, without updating model weights.The authors report two workflows and examples of resulting edits and tools.Source: E FAILURE ANALYSIS AND HARNESS… · p. 49

E.1 CRITIQUE PROTOCOL

  • Critiques use the question, answers, trajectory, images, and available tools to assign a primary failure cause and proposed fix; humans agreed with the critic on 92% of 100 sampled failures.New callable tools and text-only instruction changes are distinguished, but these labels do not directly determine which workflow to use.Source: §E.1 · p. 49

E.2 REMEDY CHANNELS AND PER-DATASET RESULTS

This section classifies critiqued failures by proposed remedy and illustrates adaptations to harness text or tools through benchmark examples and trajectories. Across visual identification, counting, and distance tasks, the examples show agents gathering and combining visual, web, depth, and coding evidence, while also exposing failed searches and grounding or coordinate errors.

E.2 REMEDY CHANNELS AND PER-DATASET RESULTS

  • Remedy channels map critic-assigned failure causes and fix types to evidence interpretation (A), output refinement (B), tool composition (C), harness text (Y), or no remedy (Z).A also includes certain text-only fixes for ignored outputs or reasoning errors despite useful evidence; this is a candidate remedy, not proof of the failure’s cause.Source: §E.2 · p. 50
  • Y accounts for 71% of ERQA and 56% of RealWorldQA failures, while A+B+C account for 75% of SparBench and 62% of FREAK-VQA failures.Among omitted-call failures, the critic proposes new tool code for 70% on SparBench but 11% on ERQA; these shares guide workflow choice, not predicted accuracy gains.Source: Table 23 · p. 51, §E.2 · p. 51

E.3 OPTIMIZATION ALGORITHMS

  • Harness search edits prompts, tool descriptions, and output templates, while tool generation iteratively builds reusable operations from existing capabilities and diagnoses their use.Generated tools must recover at least two tuning failures with positive net lift and at least one held-out failure with positive net lift; the gate tests recovery on selected baseline failures, not regressions on initially correct answers.Source: §E.3 · pp. 51–52

E.4 OPTIMIZED HARNESSES AND GENERATED TOOLS

  • The RealWorldQA text edits guide inspection of small road-scene details, count verification, and option matching, while accepted tools address SparBench geometry composition and FREAK-VQA interpretation.The critic judged 87% of Python Coding Agent calls incorrect in diagnosed SparBench failures; the visibility verifier recovered the most gate examples among the three FREAK tools, not in a benchmark ablation.Source: §E.4 · pp. 53–54

F QUALITATIVE EXAMPLES

  • The seventeen hand-selected trajectories illustrate tool composition rather than a random sample, and some are not clean wins or only partly supported by their evidence.They show inputs, answers with and without tools where available, and tool calls and outputs; the trajectories are shortened in marked ways for readability.Source: F QUALITATIVE EXAMPLES · p. 55

RESULTS

  • 139 pills were counted by GPT-6 Sol with OpenVLHarness, matching ground truth, versus 137 without tools.The trajectory combines general and round-pill grounding queries, then merges detections; after an initial silent merge failure, the coding agent revises the merge and finds five missed pills.Source: RESULTS · p. 56, F QUALITATIVE EXAMPLES · p. 56

ARGUMENTS

  • The initial mask-based merge returned zero matches and a zero union, so the agent restated the objective to compare boxes without decoding masks.The revised run found five round pills missing from the original detections.Source: ARGUMENTS · pp. 60–61

TOOL OUTPUT

  • 139 is the exact pill count, versus 137 without tools, after adding five missing round-pill detections to the 134 original detections.The second coding run found missing round detections [23, 25, 26, 40, 41] and no original detections with major overlaps.Source: TOOL OUTPUT · pp. 62–63

END OF EXAMPLE 1

  • In the OdinW-13-All PascalVOC example, the final answer matches 22 of 23 ground-truth objects in a crowded photograph of fourteen men around a dinner table.The agent first searched the four present categories, then zoomed in on the crowded table and used a referring expression for a partly hidden man.Source: END OF EXAMPLE 1 · p. 64

INPUT IMAGES

  • The PascalVOC question requests normalized bounding boxes and labels for instances in twenty named categories; the displayed evaluator uses F1 at IoU ≥0.5 for four ground-truth categories.The supplied passage gives the evaluator metric and category scope, but no per-image scores.Source: INPUT IMAGES · pp. 64–65

With OpenVLHarness: full trajectory

TOOL OUTPUT

  • In the crowded-scene example, the agent’s answer has 23 labelled boxes; the evaluator matches 22 to ground truth, yielding one false positive and one miss.The agent zooms on the crowded table and issues narrower queries for what the first pass missed.Source: TOOL OUTPUT · p. 72

ARGUMENTS

  • After grounding people and objects, the agent zooms into crowded regions and uses narrower queries; SAM 3 accepts referring expressions when category names are insufficient.In the example, it uses “person in dark shirt” and “man behind the man in white shirt” to target specific people.Source: ARGUMENTS · p. 71, TOOL OUTPUT · p. 72

END OF EXAMPLE 2

  • For fruit cups, the agent rephrases the unusual category as fruit cup, checks two regions with zooms, and returns 53 boxes that all match; SAM 3 alone returns 57 for fruit cup at 0.95.SAM 3 alone returns 25 for the original query at 0.64.Source: END OF EXAMPLE 2 · p. 73, INPUT IMAGES · p. 73

INPUT IMAGES

  • 53 fruit cups score 1.00 with GPT-6 Sol + OpenVLHarness, versus 0.00 without tools; the no-tool score reflects pixel coordinates rather than the requested 0–1000 scale.Rescaling the no-tool boxes would yield F1 1.00 and mean IoU 0.81; the harness result is attributed mainly to the coordinate frame and tighter boxes.Source: INPUT IMAGES · p. 73

TOOL OUTPUT

  • 53 returned boxes all matched ground-truth cups, after “fruit cup” produced 57 candidates versus 25 for the literal category query.The agent zoomed into two regions but the trajectory does not explain how it removed four candidates; the no-tool score of zero reflects coordinate formatting, not failure to find the objects.Source: TOOL OUTPUT · pp. 74, 76, 77, With OpenVLHarness: full trajectory · p. 74

END OF EXAMPLE 3

  • Text-based image search surfaced a gallery for the Hermès show at Paris Fashion Week Fall/Winter 2025, and a webpage visit confirmed the event.Reverse-image search found no relevant matches, and OCR read a municipal slogan on a Paris litter bag rather than a brand; the no-tool model instead guessed Miu Miu and 2024.Source: END OF EXAMPLE 3 · p. 78, INPUT IMAGES · p. 78, TOOL OUTPUT · pp. 79–81

TOOL OUTPUT

  • When reverse image search fails, the agent searches with a verbal description to find the exact street-style gallery and avoids being misled by an unrelated sign.The gallery page documents street style at Paris Fashion Week Fall/Winter 2025 at the Hermes show.Source: TOOL OUTPUT · p. 81

END OF EXAMPLE 4

TOOL OUTPUT

  • After a webpage fetch failed, text search identified the Ford Strikers Riot as occurring on April 3, 1941, completing the second evidence chain for the 9-year calculation.The first image’s event was identified from a webpage as the Ford Hunger March of March 7, 1932.Source: TOOL OUTPUT · pp. 83, 85

With OpenVLHarness: full trajectory

INPUT IMAGES

  • GPT-6 Luna changed an incorrect no-tool guess of AHC to MYONS (弥玥泉), matching ground-truth Miyue Spring.The agent zoomed and cropped the shelf before reverse-searching the crop; zoom, OCR, and whole-image search had not identified the product.Source: INPUT IMAGES · p. 87, END OF EXAMPLE 5 · p. 87

TOOL OUTPUT

  • After whole-image search fails, the agent grounds similar bottles and tightens the crop until reverse image search identifies the shelf product.The process alternates perception and search rather than continuing to search blindly.Source: TOOL OUTPUT · p. 93

END OF EXAMPLE 6

  • The foundation example’s summary reports that a tighter reverse-search crop names KATOKATO, followed by an image search returning a matching KATO-KATO product.The summary notes that the brand name first appears in the reverse-search tool’s own crop summary.Source: END OF EXAMPLE 6 · p. 94

INPUT IMAGES

  • GPT-6 Sol + OpenVLHarness answers KATO-KATO, versus Dior without tools and ground-truth KATO.The brand name first appears in the reverse-search tool’s summary, so the decisive identification is partly inside the tool.Source: INPUT IMAGES · p. 94, END OF EXAMPLE 6 · p. 94

With OpenVLHarness: full trajectory

ARGUMENTS

  • The agent rejected image-search candidates that lacked one or more requested bottle attributes, including the faceted design, silver cap, or black base.A subsequent Jill Stuart hypothesis also failed to match, while another descriptive search returned only empty stock bottles.Source: ARGUMENTS · p. 98, TOOL OUTPUT · p. 99

TOOL OUTPUT

  • The agent’s targeted search identified a KATO-KATO liquid foundation whose crystal cap matched the crop, and its final answer matched the ground truth, KATO.The agent returned to reverse image search after rejecting a Jill Stuart hypothesis and checked the candidate against the visible cap.Source: TOOL OUTPUT · pp. 101–102

END OF EXAMPLE 7

  • In the FVQA example, nine calls across six tools identify a Deuter Speed Lite Pro backpack and verify its ISPO Award 2024 on the manufacturer’s press release.The product page did not mention an award, so the agent searched for awards and checked the result against the manufacturer’s press release.Source: END OF EXAMPLE 7 · p. 103

INPUT IMAGES

  • For the backpack-award question, Qwen3-VL-32B with OpenVLHarness answered ISPO Award 2024, matching the ground truth; no no-tool run was available for comparison.The results passage explicitly notes that the question has no run of this model without tools.Source: INPUT IMAGES · p. 103

With OpenVLHarness: full trajectory

  • In the supplied trajectory, the agent grounded two backpacks and chose to zoom in on one to inspect it for identifying logos or labels.The visual-grounding tool returned bounding boxes for both backpacks; the next shown model step begins inspection of object 1.Source: With OpenVLHarness: full trajectory · p. 103, TOOL OUTPUT · p. 104

TOOL OUTPUT

  • After OCR failed on the backpacks, image search identified the green one as a Deuter Speed Lite Pro, prompting an award search and source verification.The image search returned Deuter Speed Lite Pro matches; the OCR tool reported no readable text from the purple backpack and extracted no text from the green one.Source: TOOL OUTPUT · pp. 105, 106, 109
  • The Deuter Speed Lite Pro backpack won the ISPO Award 2024 for Lightweight Speed on the Trail.The award was verified against the outdoorindustry.org press release.Source: TOOL OUTPUT · pp. 108–109

END OF EXAMPLE 8

  • The plant species in Example 9 was never identified, and the four provinces appeared in the model’s reasoning before searches supported them.The searches only confirmed that the genus is diverse in Yunnan; the passage says the answer matches the ground truth but is not fully grounded in retrieved evidence.Source: END OF EXAMPLE 8 · p. 110

INPUT IMAGES

  • Qwen3-VL-32B with OpenVLHarness answered Yunnan, Guizhou, Sichuan, and Guangxi, matching the ground truth’s provinces.The ground truth specifies Sichuan, eastern Yunnan, northern Guangxi, and Guizhou.Source: INPUT IMAGES · p. 110

TOOL OUTPUT

  • In Example 9, the agent rejects its Castanopsis argentea hypothesis after a page reports Java and Sumatra as native regions, but its final province list relies partly on an earlier guess.The page does not mention China; the text notes that retrieved pages support only part of the final answer.Source: TOOL OUTPUT · pp. 113–114

END OF EXAMPLE 9

  • For the crosswalk question, the agent combines metric depth estimation, grounding of the crosswalk and cars, and code comparing their depths.The code receives the depth map and grounded detections as data bindings in memory.Source: END OF EXAMPLE 9 · p. 115

RESULTS

  • Qwen3-VL-32B + OpenVLHarness answers B. No correctly, while Qwen3-VL-32B without tools answers A. Yes incorrectly.The harness compares the crosswalk’s 17.40-meter depth with the cars’ 54.14-meter average depth and concludes none is closer.Source: RESULTS · p. 115, TOOL OUTPUT · p. 119

END OF EXAMPLE 10

  • For the church question, whole-image searches and text descriptions return look-alikes across eleven calls; cropping the altarpiece for reverse image search finds an archive photo, and text search confirms the church.Source: END OF EXAMPLE 10 · p. 120

RESULTS

  • GPT-6 Sol + OpenVLHarness identifies St. Augustine’s Parish Church in Perchtoldsdorf, Austria, while GPT-6 Sol without tools guesses Antwerp Cathedral.The ground truth is Pfarrkirche hl. Augustinus, Perchtoldsdorf.Source: RESULTS · p. 120

With OpenVLHarness: full trajectory

TOOL OUTPUT

  • A text search returned Antwerp Cathedral, the same guess the model made without tools.The agent then searched a wide nave-and-altar crop, but reverse-image matches remained generic Gothic interiors.Source: TOOL OUTPUT · pp. 121–122
  • A German text search surfaced Münster Cathedral, but a webpage check found descriptions that did not match the photograph and subsequent searches produced no match.Another descriptive search returned stock altar photos.Source: TOOL OUTPUT · pp. 124–125
  • The agent’s key next step was a tight crop of the altarpiece alone, isolating the detail unique to the church.Source: TOOL OUTPUT · p. 126

HARNESS

  • Cropping the altarpiece makes reverse-image search specific: its first result matches the altar painting, and text search confirms the church.Searching the whole photograph yields look-alike interiors; earlier text searches, page visits, and a wide crop do not narrow them down.Source: TOOL OUTPUT · pp. 126–127
  • The agent identifies St. Augustine’s Parish Church in Perchtoldsdorf, Austria, and the judge accepts its answer.Cropping the altarpiece enables a specific reverse-image match, which a text search then confirms.Source: HARNESS · p. 127

RESULTS

  • GPT-6 Sol answers “left” correctly with OpenVLHarness, while its no-tool answer “right” is incorrect.The ground truth is left.Source: RESULTS · p. 128

TOOL OUTPUT

  • After three zooms, grounding distinguishes the backpacked woman from a second person to her right, supporting the answer that she is on the left.The beanie query also returns the woman’s box, so the final judgment still relies on the model’s reading of the crop.Source: END OF EXAMPLE 11 · p. 128, TOOL OUTPUT · p. 133

END OF EXAMPLE 12

  • For a seven-photo living-room task, the agent grounds pillows across four seating views, zooms on both sofa halves, and chooses the statement giving 16 pillows.The task involves partly hidden and repeated pillows; without tools, the model picks 12.Source: END OF EXAMPLE 12 · p. 134

RESULTS

  • C (16 throw pillows) with OpenVLHarness is correct, while GPT-6 Sol without tools answers D (12 throw pillows).The ground truth is C.Source: RESULTS · p. 134

With OpenVLHarness: full trajectory

  • Grounding found 18 pillow detections across four views; after zooming on the sofa, the agent implicitly deduplicated overlapping views and answered 16.The agent does not write out which detections it treated as duplicates.Source: TOOL OUTPUT · p. 139

END OF EXAMPLE 13

  • In the piano example, the harness answers correctly, but compares the piano with the round dining table instead of the mantel shelf under the mirror.It concludes the table is 2.77 m and the piano 2.21 m; the passage says this evidence does not support the answer.Source: END OF EXAMPLE 13 · p. 140, INPUT IMAGES · p. 140

ARGUMENTS

  • The code estimates the piano at 2.2109 meters and the dining table at 2.7731 meters, concluding that the table is longer.It combines bounding-box widths with depth values sampled at object centers using a projected-length formula.Source: ARGUMENTS · p. 144
  • The piano-and-table example shows a failure mode: the model selects the wrong grounding result, and no later step checks that choice.The model composes grounding, depth, and code to make the comparison, stating its plan before the first tool call.Source: ARGUMENTS · p. 145

END OF EXAMPLE 14

  • The SparBench example asks for the distance between two marked bookshelves that never appear together in a photo; without tools, GPT-6 Sol answers 4.0 m.The harness estimates the distance at 2.0 m against 1.9 m ground truth.Source: END OF EXAMPLE 14 · p. 146

INPUT IMAGES

  • GPT-6 Sol + OpenVLHarness gives 2.0 m (MRA 0.90), closer to the 1.9 m ground truth than GPT-6 Sol without tools at 4.0 m (MRA 0.00).Source: INPUT IMAGES · p. 146

With OpenVLHarness: full trajectory

  • For the bookshelf measurement, the trajectory tool estimates that the third photo was taken about 1.5 m left and 1.65 m behind the first.The resulting camera parameters are retained as a data binding for code.Source: With OpenVLHarness: full trajectory · p. 146

TOOL OUTPUT

  • Camera trajectory estimates and metric depths let the coding agent transform points from separate photos into one 3D frame and measure their distance.For the bookshelf example, the point depths are 2.15 m and 4.08 m, and the computed distance is 2.0076 m.Source: With OpenVLHarness: full trajectory · p. 146, TOOL OUTPUT · p. 149
  • For the preceding cross-view example, the harness combines camera poses, metric depths, and code to compute a 2.0076 m distance between points not visible in the same image.The agent rounds the computed distance to 2.0 m.Source: TOOL OUTPUT · pp. 150–151

END OF EXAMPLE 15

  • 2.4 m achieves 1.00 MRA and matches ground truth, versus 2.0 m and 0.00 MRA without tools.The task asks for the distance from the observer’s position to a cabinet visible only in the third photo.Source: INPUT IMAGES · p. 152, END OF EXAMPLE 15 · p. 152

With OpenVLHarness: full trajectory

ARGUMENTS

  • A coding attempt to read the cabinet depth first fails on an undefined math import; a narrower retry obtains 2.16 m from the third camera, not the observer.The retry computes a local median depth of 2.1564598 m at the marked point.Source: ARGUMENTS · p. 154, TOOL OUTPUT · pp. 154–155

END OF EXAMPLE 16

  • Transforming the marked point from the third camera into the observer’s frame yields 2.3995 m, which the agent reports as 2.4 m and matches ground truth.The computation unprojects the marked pixel using metric depth, then applies the camera-trajectory extrinsics.Source: TOOL OUTPUT · pp. 156–157
  • In Example 16, code misread corner-coordinate boxes and selected an upper wall cabinet instead of the marked lower cabinet, making the 1.20 m answer close to 1.1 m partly by coincidence.The tool pipeline was appropriate, but the final computation did not use the marked points.Source: END OF EXAMPLE 16 · p. 158

INPUT IMAGES

  • Qwen3-VL-32B + OpenVLHarness answered 1.1957 against a 1.1 ground truth (MRA 0.90), versus 1.7 without tools (MRA 0.00).The question asks for the 3D distance between the red-marked window center and blue-marked cabinet center across views.Source: INPUT IMAGES · p. 158

With OpenVLHarness: full trajectory

TOOL OUTPUT

  • In the shown trajectory, the first coding attempt failed because a pixel index exceeded the depth-map bounds, and the next attempt instructed the agent to clamp coordinates.The supplied passages do not show whether this correction succeeded.Source: TOOL OUTPUT · p. 163, ARGUMENTS · p. 163

ARGUMENTS

  • The example reports a 1.1957-meter window–cabinet distance, but the code measures from the wrong cabinet, so agreement with ground truth is not evidence of success.The tool chain uses camera poses, depth from each view, object grounding, and code to compute a world-coordinate distance; the model plans before its first call.Source: ARGUMENTS · p. 166

G BASELINE REPRODUCTION

  • Baselines receive the same questions, answer-format instructions, scoring pipeline, and full evaluation split, with failures or timeouts counted as incorrect.Qwen3-VL OctoTools and Visual Sketchpad use the same three-run protocol as OpenVLHarness; proprietary-backbone and training-based baseline runs are single runs.Source: G BASELINE REPRODUCTION · p. 166
  • Table 25 lists reproduced-baseline configurations and indicates which methods have web search; all but Codex decode greedily.Codex uses its own request settings.Source: Table 25 · p. 167

G.1 TRAINING-FREE HARNESSES

The section describes how training-free and training-based visual tool-use baselines were configured and evaluated. It standardizes comparison conditions while preserving each baseline’s released inference approach.

G.1 TRAINING-FREE HARNESSES

  • Training-free harnesses use the same model endpoints as OpenVLHarness, while OctoTools and Visual Sketchpad retain their respective planner–executor and ReAct-style workflows.OctoTools uses a ten-step budget; Visual Sketchpad uses up to ten replies.Source: §G.1 · p. 167
  • Codex runs in fresh, non-interactive sessions with external browsing disabled, while its sandbox and execution limits vary by backbone.GPT-6 Luna runs use a read-only policy; Qwen3-VL runs use the default sandbox and are capped at 2,048 tokens per response.Source: §G.1 · p. 167

G.2 TRAINING-BASED HARNESSES

  • Training-based baselines run their released checkpoints in verified author inference loops, with a lightweight serving layer standardizing requests and returned predictions.A shared external harness performs scoring and compares these methods with OpenVLHarness on their common Qwen2.5-VL-7B-Instruct base model.Source: §G.2 · p. 168
  • The training-based baselines cover adaptive tool composition, pixel-space reasoning, and agentic code execution with retrieval.AdaReasoner uses up to six rounds; PixelReasoner preserves its documented image-resolution budgets; DeepEyesV2 binds ordered images in its Python sandbox.Source: §G.2 · p. 168
  • Fair-comparison controls cover datasets, prompts, model settings, tool access, concurrency, judging, and failure handling.Source: §G.2 · p. 168

H RUN-TO-RUN VARIABILITY

The section quantifies run-to-run variability and compares it with method and ablation differences, then situates OpenVLHarness among related work on multimodal search, perception, spatial reasoning, and harness optimization.

H RUN-TO-RUN VARIABILITY

  • Qwen3-VL results average three independent runs, whereas proprietary-backbone results are single runs; three-run standard deviations are coarse indicators of variability.Live search also introduces variation from changes in retrieved content.Source: H RUN-TO-RUN VARIABILITY · p. 169
  • +11.2 points at 8B and +12.6 at 32B for OpenVLHarness over Base are far larger than observed Qwen3-VL run variation; proprietary-backbone gains are also large relative to that variation.The reported proprietary-backbone gains are +10.8 with GPT-6 Luna, +11.6 with GPT-6 Sol, and +9.9 with GPT-6 Astra.Source: H RUN-TO-RUN VARIABILITY · p. 169
  • Ablation averages shift by several points when tools, the capability layer, or multimodal memory are added, while textual-memory gains are comparable to run-to-run variation.The textual-memory step is +0.7 at 8B and +0.3 at 32B.Source: H RUN-TO-RUN VARIABILITY · p. 169

I ADDITIONAL RELATED WORK

  • Multimodal search benchmarks span curated search, provenance verification, cross-modal information seeking, visual-native search, and perception with multi-hop retrieval; OpenVLHarness generates rather than freezes its tool inventory.Source: I ADDITIONAL RELATED WORK · pp. 170–171
  • For grounded perception, OpenVLHarness retains boxes, masks, and scores as typed evidence that code can filter, deduplicate, and aggregate across turns.Source: I ADDITIONAL RELATED WORK · p. 171
  • For spatial reasoning, geometric foundation models are callable capabilities whose outputs can be composed with grounding and other persistent evidence.Source: I ADDITIONAL RELATED WORK · p. 171
  • Prior harness optimization methods rewrite prompts, pipelines, or harness code from observed failures, but the cited work operates in the text-only prompt and harness domain.Source: I ADDITIONAL RELATED WORK · p. 171
Loading 2610.26860v1…