PaperlayerView exact revision on arXiv

Source-linked AI summary

VipAct: Visual-Perception Enhancement via Specialized VLM Agent Collaboration and Tool-use

Zhehao Zhang, Ryan Rossi, Tong Yu, Franck Dernoncourt, Ruiyi Zhang, Jiuxiang Gu, Sungchul Kim, Xiang Chen, Zichao Wang, Nedim Lipka

arXiv:2410.16400v2cs.CL

TL;DR

VLMs remain challenged by fine-grained visual perception that calls for pixel-level analysis and detailed reasoning. VipAct combines an orchestrator, specialized agents, and vision expert models; it reports superior performance across diverse tasks and benchmarks. Ablations highlight multi-agent collaboration and image input for task planning, while error analysis identifies recurring perception errors.

  • Problem

    VLMs continue to struggle with fine-grained visual perception requiring detailed pixel-level analysis and comprehensive reasoning.

    Source: §1 · p. 1

  • Method

    VipAct uses an orchestrator to analyze and plan tasks, specialized agents for focused visual analysis, and vision expert models for fine-grained perceptual information.

    Source: §1 · p. 1, §3 · p. 2

  • Results

    VipAct consistently outperforms previous baselines across diverse visual perception tasks and benchmarks.

    Source: §1 · p. 2, §7 · p. 7

  • Takeaways & Limitations

    Ablations indicate that multi-agent collaboration supports more detailed reasoning and direct image input is important for flexible task planning and error handling.

    Source: §5 · pp. 6–7

    Fine-grained spatial reasoning is a prominent error category (24%), including a tendency to favor foreground objects over backgrounds.

    Source: §6 · p. 7

Abstract

from arXiv · show

While vision-language models (VLMs) have demonstrated remarkable performance across various tasks combining textual and visual information, they continue to struggle with fine-grained visual perception tasks that require detailed pixel-level analysis. Effectively eliciting comprehensive reasoning from VLMs on such intricate visual elements remains an open challenge. In this paper, we present VipAct, an agent framework that enhances VLMs by integrating multi-agent collaboration and vision expert models, enabling more precise visual understanding and comprehensive reasoning. VipAct consists of an orchestrator agent, which manages task requirement analysis, planning, and coordination, along with specialized agents that handle specific tasks such as image captioning and vision expert models that provide high-precision perceptual information. This multi-agent approach allows VLMs to better perform fine-grained visual perception tasks by synergizing planning, reasoning, and tool use. We evaluate VipAct on benchmarks featuring a diverse set of visual perception tasks, with experimental results demonstrating significant performance improvements over state-of-the-art baselines across all tasks. Furthermore, comprehensive ablation studies reveal the critical role of multi-agent collaboration in eliciting more detailed System-2 reasoning and highlight the importance of image input for task planning. Additionally, our error analysis identifies patterns of VLMs' inherent limitations in visual perception, providing insights into potential future improvements. VipAct offers a flexible and extensible framework, paving the way for more advanced visual perception systems across various real-world applications.

1 Introduction

VLMs still struggle with fine-grained visual perception, including line intersections and object boundaries, despite broader progress on image-text tasks. VIPACT combines multi-agent collaboration with vision expert models and reports consistent gains across diverse benchmarks.

1 Introduction

  • VLMs struggle with fine-grained tasks such as detecting line intersections and object boundaries, despite strong performance on broader image-text benchmarks.Precise visual understanding is needed in applications such as surgical robotics and autonomous driving.Source: §1 · p. 1
  • VIPACT combines multi-agent collaboration and vision expert models to improve fine-grained visual perception.Its orchestrator coordinates specialized agents and expert models; experiments report gains over state-of-the-art baselines across diverse tasks.Source: §1 · pp. 1–2
  • VIPACT uses a single prompt template and plug-and-play modules to handle diverse visual perception tasks autonomously.The framework combines VLM-based task analysis and planning with multi-agent collaboration.Source: §1 · p. 2
  • The paper evaluates VIPACT across diverse benchmarks and reports advantages over state-of-the-art baselines, alongside ablations and error analysis.Its analyses examine collaboration, image input for planning, and limitations of current VLMs.Source: §1 · p. 2

2 Related Work

Prior work on visual agents has focused largely on web and GUI interaction or robotic control, while agents for natural-image perception remain less explored. Visual-programming systems use tools for visual reasoning but are constrained by rigid tool selection and limited task coverage.

VLM-based Agent

  • VLM-based agents for natural-image perception remain unexplored compared with work on web/GUI agents and embodied robot control.The related-work discussion identifies these as the main existing application areas for vision-centric agents.Source: §2 · p. 2

Visual Programming

  • Visual-programming systems such as MM-REACT, ViperGPT, and VisProg use LLMs with vision experts or generated code, but often rely on text queries and rigid tool selection.These constraints limit adaptation to new tasks and support for fine-grained perception, visual prompts, and multi-image inputs.Source: §2 · p. 2
  • Existing visual-programming methods are described as having limited support for complex visual reasoning beyond simpler visual question answering.The passage notes that their utility is restricted by their lack of fine-grained perception, visual-prompt, and multi-image support.Source: §2 · p. 2

3 VipAct Framework

VipAct coordinates an orchestrator, specialized agents, and vision expert models to combine task planning with detailed visual evidence. Its workflow decomposes tasks, selects tools as needed, and integrates their outputs into a final answer.

3 VipAct Framework

  • VipAct's three components are an orchestrator that plans and coordinates, specialized agents that analyze images, and vision expert models that supply fine-grained perceptual information.Not every agent or expert model is invoked for every task.Source: §3 · p. 2, Figure 1 · p. 3

3.1 Orchestrator Agent

  • The orchestrator analyzes task requirements, plans the needed visual information, selects tools or agents, and synthesizes evidence before deducing an answer.It resolves conflicting evidence and checks factuality because tools and agents may produce errors or hallucinations.Source: §3.1 · pp. 2–3

3.2 Collaboration with Specialized Agents

  • Specialized agents provide focused image captions, visual-prompt descriptions, and image comparisons, with an extensible design for adding agents.Their analyses can target task-relevant image elements or compare specific features across images.Source: §3.2 · p. 4

3.3 Integration of Vision-Expert Models

  • Vision expert models provide task-specific pixel-level evidence, including prompt locations, depth maps, object boxes, segmentation masks, and image-similarity scores.The orchestrator dynamically selects among tools, and processed images may accompany textual outputs in its reasoning workflow.Source: §3.3 · p. 4

4 Experiment

Experiments evaluate VIPACT on Blink and MMVP with closed- and open-source VLMs and compare it with prompting, visual-prompting, and agentic baselines. VIPACT is reported to lead across the benchmark tasks, while prior prompting and visual-programming approaches show inconsistent coverage or performance.

Datasets

  • The evaluation uses Blink, which includes visual prompts and multi-image queries, and MMVP, which tests visual grounding across nine basic patterns.Blink is the main benchmark; both datasets target fine-grained perception that is challenging for state-of-the-art VLMs.Source: §4 · p. 5

Baselines

  • VIPACT is compared with text-prompting, few-shot, visual-prompting, and vision-language agentic baselines.Experiments include GPT-4o, Gemini-1.5-Pro, Claude-3.5-Sonnet, LLaVA-OneVision-7B, and other models; the paper focuses on GPT-4o in its main analysis.Source: §4 · pp. 4–5

Result Analysis

  • Text-based prompting methods do not consistently improve over zero-shot prompting, and Set-of-Mark can impair fine-grained perception in most scenarios.The authors attribute some prompting failures to reasoning ungrounded in visual elements and mark overlays that clutter images.Source: §4 · p. 5
  • VIPACT consistently achieves the best performance across Blink and MMVP subtasks, aggregating useful tool evidence with additional reasoning when outputs may be incorrect.Table 2 reports Blink results with GPT-4o; the paper's analysis also reports best performance across MMVP subtasks.Source: Table 2 · p. 6, Table 3 · p. 6, §4 · p. 6

5 Ablation Study

The ablations test multi-agent collaboration, direct image input to the orchestrator, specialized agents, and vision expert models. Removing collaboration or direct image input degrades performance, and the authors report that specialized agents and vision experts also contribute.

5 Ablation Study

  • Removing multi-agent collaboration consistently lowers performance and reduces generated reasoning tokens by over 80%.The authors observed less detailed image analysis, including less thorough image captioning.Source: §5 · p. 6, Table 4 · p. 7, Table 5 · p. 7
  • Removing direct image input to the orchestrator significantly degrades performance and can lead to less precise parameter selection and less specific task analysis.The authors give identifying reference objects in a multi-view task as an example of how direct image input can inform focus and camera-movement choices.Source: §5 · pp. 6–7, Table 4 · p. 7, Table 5 · p. 7
  • Removing specialized agents or vision expert models leads to a noticeable performance decline, with experts supplying pixel-level analyses beyond VLM capabilities.Specialized agents focus on specific visual information without distractions from other instructions.Source: §5 · pp. 6–7, Table 4 · p. 7, Table 5 · p. 7

6 Error Analysis

The error analysis samples 20 cases per subtask from two datasets and categorizes recurring visual-perception errors. It finds that spatial reasoning remains a bottleneck, particularly for real-life scenes, visual prompts, and multiple-image tasks.

6 Error Analysis

  • The analysis sampled 20 error cases from each subtask across two datasets and identifies spatial reasoning as a bottleneck, especially for real-life scenes, visual prompts, and multiple-image tasks.The authors note difficulty inferring object relations beyond camera-view pixel positions.Source: §6 · p. 7
  • 24% of sampled errors involved fine-grained spatial reasoning, including associating a sky highlight near a car with the car.The analysis describes a bias toward foreground objects over backgrounds.Source: §6 · p. 7
  • 17% of sampled errors involved overlooking small, semantically important object parts, while 15% involved distinguishing closely positioned visual prompts.Source: §6 · p. 7
  • 14% of errors involved relative object positions, 13% involved object orientation, and 17% fell into other categories.Other errors included subtle color differences, multi-image structure correspondence, refusals, and instruction misinterpretation.Source: §6 · p. 7

7 Conclusion

VIPACT combines multi-agent collaboration with vision expert models to address fine-grained visual-perception tasks. The paper reports state-of-the-art performance across diverse tasks, while ablations and error analysis identify contributions and remaining limitations.

7 Conclusion

  • VIPACT combines VLM planning and function calling with multi-agent interactions and specialized vision models that provide high-precision, pixel-level information.Source: §7 · p. 7
  • Experiments across a diverse range of visual-perception tasks show VIPACT outperforming previous baselines and achieving state-of-the-art performance.Source: §7 · p. 7
  • Ablations identify multi-agent collaboration as important for eliciting detailed reasoning and direct image input as important for task planning.Source: §7 · p. 7
  • Error analysis identifies inherent limitations in current state-of-the-art VLMs as bottlenecks for VIPACT and points to areas for future improvement.Source: §7 · p. 7

A Implementation Details

The experiments use GPT-4o for VIPACT and specify vision-expert checkpoints and baseline implementations. Runs are averaged across three trials, with reported significance testing and inference-time and hardware details.

A Implementation Details

  • Main experiments use gpt-4o-2024-05-13 through Azure OpenAI, with temperature 0 and a 2048-token maximum for VLM inference.The same model implements the orchestrator and specialized agents.Source: A Implementation Details · p. 13
  • Vision experts use Depth-Anything-V2-Small-hf for depth estimation, SAM for segmentation, YOLOv8 for object detection, and CLIP ViT-B/32 for cosine-similarity comparison.Source: A Implementation Details · p. 13
  • Experiments are run three times and averaged; the reported average p-value for VIPACT versus state-of-the-art baselines on Tables 2 and 3 is 0.006 (< 0.01).Source: A Implementation Details · p. 13
  • Baseline implementations use original codebases and settings where specified, with GPT-4o replacing unavailable backbones for MM-ReAct, ViperGPT, and VisProg.VIPACT inference on Blink and MMVP takes less than two hours on a Linux server with four NVIDIA A100-40G GPUs.Source: A Implementation Details · p. 13

B Dataset Details

The experiments use Blink and MMVP to evaluate distinct aspects of visual perception, excluding Blink tasks centered on compositional reasoning.

B Dataset Details

  • Blink evaluation focuses on ten sub-tasks spanning similarity, counting, depth, puzzles, correspondence, spatial relations, and multi-view reasoning.The passage also refers to validation and test splits, with task-level data-point counts summarized in Table 6.Source: B Dataset Details · p. 13
  • Blink datasets focused on IQ testing or commonsense reasoning are excluded because they emphasize compositional reasoning rather than visual perception.Source: B Dataset Details · p. 13
  • MMVP comprises 150 CLIP-blind image pairs and 300 questions probing nine visual patterns, including orientation, quantity, color, text, and viewpoint.Human participants achieved 95.7% accuracy, while GPT-4V and Gemini performed significantly worse.Source: B Dataset Details · p. 13

C Exploration of different VLMs

VIPACT improves results over prompting baselines with close-source models that follow instructions, while open-source models show weaker and less consistent performance. Case studies illustrate both successful tool-and-agent coordination and persistent fine-grained reasoning errors.

C Exploration of different VLMs

  • Open-source VLMs struggle on complex or non-standard VQA: LLaVA-OneVision-7B is near or below random on many tasks, while InternVL-2-Pro and Llama-3.2-90b-vision remain behind GPT-4o.Alternative prompting strategies yield no noticeable gains over zero-shot baselines, and LLaVA-OneVision-7B shows positional bias toward the first option.Source: C Exploration of different VLMs · p. 13
  • VIPACT consistently outperforms previous prompting baselines when applied to Gemini-1.5-Pro and Claude-3.5-Sonnet, whose instruction following is comparable to GPT-4o.The framework's effective application to these models depends on their instruction-following ability.Source: C Exploration of different VLMs · pp. 13–14

D Case Studies

  • VIPACT's specialized agents can still make reasoning errors that mislead the orchestrator despite its evidence-reassessment mechanism.Examples include misreading a bicycle pedal's orientation, overlooking a cap brim, and misjudging relative height from camera perspective.Source: D Case Studies · p. 14
  • In the depth case, the orchestrator combines a Visual Prompt Detector with a Depth Estimator to identify the depth values of two red points, where GPT-4o alone cannot perceive depth.In another case with no directly applicable vision tool, a visual-prompt-description agent analyzes marked regions and supports the correct prediction.Source: D Case Studies · p. 14

E Few-shot In-context Learning

Few-shot prompting offers no consistent advantage across the reported settings, while concatenating multiple images reduces performance. The section also outlines VIPACT's algorithm and function definitions for coordinating agents and vision experts.

E Few-shot In-context Learning

  • Few-shot exemplars are appended as image-question-answer triplets, selected randomly or by top-K similarity using CLIP or ViT embeddings.Source: E Few-shot In-context Learning · p. 14
  • GPT-4o few-shot prompting can reduce performance on counting and spatial relations, while LLaVA-OneVision-7B remains near random on almost all Blink sub-tasks and retains or worsens its positional bias.Across GPT-4o tasks, increasing shots gives no consistent trend, and few-shot prompting offers no noticeable benefit to LLaVA-OneVision-7B.Source: E Few-shot In-context Learning · p. 14
  • Table 7 reports Blink visual-reasoning results for Gemini-1.5-Pro and Claude-3.5-Sonnet, with VIPACT outperforming baselines on almost all tasks.Source: Table 7 · p. 15

F Exploring the Importance of Multiple Image Inputs to VLMs

  • Concatenating multiple images into one causes a noticeable performance decline for GPT-4o and LLaVA-OneVision-7B on Blink tasks requiring multiple image inputs.The decline is particularly consistent for GPT-4o; the authors conclude concatenation is not ideal and native multiple-image support is important for maintaining performance.Source: F Exploring the Importance of Multiple… · pp. 15–16, Table 10 · p. 16

G Detailed VIPACT Algorithm and Function Definitions

H In-depth Analysis of Visual Input

This section analyzes VipAct’s visual-input and prompt-design choices, including direct images versus text descriptions, iterative tool use, focused analysis of highlighted regions, and few-shot examples.

H In-depth Analysis of Visual Input

  • Text-only descriptions underperform image input; combining both yields only slight gains, as descriptions can omit contextual and background details needed for spatial analysis and precise planning.The study compared image-only, GPT-4o-generated description-only, and combined image-plus-description inputs.Source: H In-depth Analysis of Visual Input · p. 16, Table 13 · p. 18

I Analysis of Tool Set Fairness in Comparisons

  • Enhanced baselines gain only slightly on depth estimation and object localization from VIPACT’s tools, yet remain significantly below VIPACT because of limitations in their underlying logic.VIPACT’s modular design allows tool integration by defining a standard Python function header while preserving each baseline’s original logic.Source: I Analysis of Tool Set Fairness in… · p. 16, Table 14 · p. 18

J Prompt Design

  • The orchestrator prompt requires task decomposition, tool and parameter selection, iterative thought-action-observation, and a clear final answer.It directs the agent to analyze task requirements and use tool observations to inform subsequent reasoning.Source: J Prompt Design · p. 25
  • Focused captioning and comparison prompts request detailed descriptions of focus-element appearance, quantity, position, relations, context, and cross-image similarities and differences.The prompts also call for relevant actions, events, lighting, shadows, and textures.Source: J Prompt Design · p. 26
  • The visual-prompt agent is instructed to prioritize detailed descriptions of the exact regions or objects indicated by circles, boxes, arrows, highlights, or labels.Source: J Prompt Design · p. 27
  • The instructions call for describing a region’s precise location and appearance, including color, texture, shape, size, distinctive features, and relationships to surrounding elements.Descriptions should be detailed enough to identify the emphasized part without seeing its visual prompt.Source: J Prompt Design · p. 27
  • For each visual prompt, the instructions ask respondents to identify its type, position in the image, and any associated text or identifiers.Source: J Prompt Design · p. 27
  • The few-shot template presents example images with questions and answers before a final image-and-question prompt for the response.Source: J Prompt Design · p. 27

Related papers

Loading 2410.16400v2…