PaperlayerView exact revision on arXiv

Source-linked AI summary

V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs

Penghao Wu, Saining Xie

arXiv:2312.14135v2cs.CV

TL;DR

Current multimodal LLMs can miss important details in high-resolution, visually crowded images because they lack visual search. This paper introduces SEAL with the LLM-guided V* mechanism and reports that visual search greatly improves performance on challenging visual grounding tasks, though substantial room remains for improvement.

  • Problem

    Current MLLMs depend on low-resolution vision encoders that may overlook important details in high-resolution images.

  • Method

    SEAL integrates an LLM-guided visual search algorithm that identifies needed details, locates them, and adds them to visual working memory.

  • Results

    Visual search greatly improves performance on V*Bench, although MLLMs remain considerably below near-perfect human performance.

  • Takeaways & Limitations

    V*Bench highlights visual search as a critical capability for MLLMs processing and grounding detailed information in high-resolution images.

Abstract

from arXiv · show

When we look around and perform complex tasks, how we see and selectively process what we see is crucial. However, the lack of this visual search mechanism in current multimodal LLMs (MLLMs) hinders their ability to focus on important visual details, especially when handling high-resolution and visually crowded images. To address this, we introduce V*, an LLM-guided visual search mechanism that employs the world knowledge in LLMs for efficient visual querying. When combined with an MLLM, this mechanism enhances collaborative reasoning, contextual understanding, and precise targeting of specific visual elements. This integration results in a new MLLM meta-architecture, named Show, sEArch, and TelL (SEAL). We further create V*Bench, a benchmark specifically designed to evaluate MLLMs in their ability to process high-resolution images and focus on visual details. Our study highlights the necessity of incorporating visual search capabilities into multimodal systems. The code is available https://github.com/penghao-wu/vstar.

1. Introduction

The introduction identifies visual search as central to human visual reasoning but underdeveloped in current MLLMs, whose vision-encoder dependence limits access to important details. It presents SEAL, the LLM-guided V* search mechanism, and V*Bench for high-resolution visual grounding evaluation.

  • Motivation: Human visual search efficiently recognizes and localizes key objects in intricate real-world scenes, supporting complex interaction with the environment.The paper describes visual search as a salient aspect of cognitive reasoning involving visual information.
  • Motivation: Current MLLMs aim to integrate multimodal information and perform general tasks but depend on pre-trained, often frozen vision encoders such as CLIP.The introduction frames this dependency as a key limitation of current MLLM paradigms.
  • SEAL: SEAL integrates a VQA LLM with a visual search model so the system can explicitly pinpoint needed visual details rather than refuse or guess when encoder information is insufficient.SEAL stands for Show, SEArch, and TelL and is designed as a general MLLM meta-architecture.
  • V*: V* uses an MLLM’s common-sense world knowledge to reason about likely target locations and perform efficient informed searches across images of any resolution.Its design follows human top-down feature guidance and contextual scene guidance principles.
  • V*Bench: V*Bench evaluates detailed visual grounding on high-resolution images, targeting limitations that existing MLLM benchmarks do not adequately expose.The benchmark is introduced as a dedicated VQA evaluation requiring accurate grounding of visual information.

2. Related Work

Prior visual-search models imitate human gaze using saliency, scene priors, or target-template similarity, but often use fixed-size windows and limited categorical or contextual information. MLLMs mainly comprise end-to-end models or LLM tool-using systems, while vision-feature projection can create an information bottleneck that loses visual detail.

  • Computational Visual Search: Earlier visual-search models combine saliency priors, global scene context, or target-template similarity to guide search.Sclar et al. use a Bayesian searcher with a saliency prior; Torralba et al. combine local saliency with global scene priors; IVSN computes image-template similarity.
  • Computational Visual Search: These models mainly mimic human gazing trajectories rather than accurately localizing target objects, and typically use fixed-size gazing windows.The paper contrasts these limitations with a hierarchical visual-search process that handles images at any resolution.
  • Computational Visual Search: V* uses LLM common-sense knowledge to expedite visual search, with an active strategy linked to System II cognitive processes for complex tasks.Its search process is described as dynamically allocating computation for visual search.
  • Multimodal Large Language Models: MLLMs are categorized into end-to-end models and LLM tool-using systems.End-to-end systems connect pretrained LLMs with vision encoders through projection or alignment modules to process visual information and perform vision-language tasks.
  • Multimodal Large Language Models: Projecting visual features into language or intermediate feature spaces introduces an information bottleneck and can cause visual information loss.Vision encoders such as CLIP translate visual features into language-like tokens, but conversion and projection can lose information, particularly because vision encoders are typically limited in resolution.

3. Method

SEAL is a general MLLM meta-architecture in which a VQA LLM and visual search model collaborate through visual working memory. Its V* search mechanism combines MLLM-guided contextual cues, localization decoders, and recursive image partitioning to find targets in high-resolution images.

  • SEAL Framework: SEAL combines a VQA LLM and visual search model through a visual working memory, instantiated here with LLaVA-7B.The visual working memory stores the question, global image, and searched targets.
  • SEAL Framework: The visual search mechanism activates only when global image features are insufficient, then identifies missing target objects and searches for them.The model first tests whether the encoder’s global features suffice before creating target-object queries in visual working memory.
  • Visual Search Model: V* uses an MLLM for contextual and top-down guidance plus a localization module with target and search-cue decoders.The MLLM emits a localization token, while the decoders produce target coordinates and search-cue heatmaps.
  • Visual Search Algorithm: If direct localization fails, V* uses a target-specific heatmap when available; otherwise, the MLLM predicts the target’s likely location from image context and common sense.A heatmap cue is used when its highest value exceeds threshold δ; otherwise, the MLLM supplies a contextual cue.
  • Visual Search Algorithm: The search recursively divides images into four equal-sized patches, adjusting orientation-based splits to preserve square-like patch shapes.Landscape images are divided vertically when width exceeds twice height, portrait images horizontally when height exceeds twice width, and other images are split normally.

4. Benchmark

V∗Bench evaluates detailed visual analysis in complex, high-resolution images through attribute recognition and spatial relationship reasoning. On this benchmark, SEAL achieves 74.78% attribute accuracy, 76.31% spatial accuracy, and 75.39% overall accuracy, outperforming leading systems according to the supplied comparison.

  • Benchmark construction: V∗Bench contains 191 high-resolution SA-1B images averaging 2246×1582 resolution, targeting scenarios with abundant information and difficult-to-find visual details.The benchmark is designed to quantitatively evaluate MLLMs under visually complex conditions.
  • Benchmark tasks: The benchmark comprises attribute recognition with 115 samples and spatial relationship reasoning with 76 samples, both testing detailed visual analysis.Attribute questions concern object properties such as color or material, while spatial questions ask for relative relationships between two objects.
  • Results: 74.78% attribute accuracy, 76.31% spatial accuracy, and 75.39% overall accuracy are reported for SEAL on V∗Bench.These are the reported SEAL (Ours) results for the Attribute, Spatial, and Overall metrics, respectively.
  • Results: SEAL outperforms leading-edge systems such as GPT-4V and Gemini by a large margin despite using only a Vicuna-7B LLM.The supplied table caption interprets this result as demonstrating the importance of integrating visual search into MLLMs.
  • Evaluation protocol: For open-source MLLM comparison, each question uses carefully reviewed multiple-choice options: four for open-ended questions and two for binary questions.Human annotators review the options for potential ambiguity.

5. Experiments

Experiments evaluate SEAL against end-to-end MLLMs and tool-using systems on V∗Bench, showing that existing models often struggle with its visual-detail tasks. Ablations and broader benchmarks support the effectiveness of visual search while largely preserving general multimodal capabilities.

  • Experimental setup: SEAL uses Vicuna-7B as its VQA language model and compares against open-source end-to-end MLLMs and LLM-tool-using systems on V∗Bench.End-to-end models are evaluated by selecting the answer choice with the highest log-likelihood.
  • Main results: Most evaluated MLLMs perform only near random guessing on V∗Bench, while GPT-4V and Gemini handle some easier attribute-recognition scenarios but remain unsatisfactory overall.LLaVA-1.5 substantially improves over the initial LLaVA on attribute recognition, potentially partly because it uses the higher-resolution CLIP-ViT-L-336px encoder.
  • Ablation studies: Ablations compare a LLaVA model without VWM against versions replacing visual search with GroundingDINO or OWL-ViT, testing the necessity of the proposed mechanism.All variants are trained or evaluated using the same underlying VQA setup described in the ablation experiment.
  • Search strategy: V∗ greatly reduces average search length relative to random and sequential baselines, with both target-specific and contextual search cues contributing useful guidance.Search length counts steps from the initial image to the target patch, and evaluation includes successfully located samples among 245 recorded target locations.
  • General multimodal capability: Adding visual search improves MME and alleviates hallucinations on POPE, largely preserves MMBench and SEED-Bench performance, and slightly lowers MM-Vet performance.The comparison uses a LLaVA model trained on the same VQA training data.

6. Conclusion

The paper introduces SEAL, an MLLM framework that uses the LLM-guided V* algorithm for accurate visual grounding in high-resolution images, and V*Bench to assess visual search capabilities. The current approach is tailored to natural images and common objects, requiring further training and algorithmic development for broader settings.

  • 6. Conclusion: SEAL combines an MLLM with V* for accurate visual grounding in high-resolution images.V* is an LLM-guided visual search algorithm.
  • 6. Conclusion: V*Bench highlights the critical role of visual search capabilities in MLLMs.The benchmark evaluates the importance of visual search within multimodal systems.
  • 6. Conclusion: The current visual search model is primarily tailored to natural images and common objects.Extending it to document and diagram images, long-form videos, or open-world environments requires additional training and new algorithm design.

A. Implementation Details · A.1. Data Curation for VQA LLM

The implementation curates VQA data by selecting questions tied to annotated visual objects and attributes, then constructing additional object-focused examples from LLaVA-80K instructions. The process uses model-based answerability checks, object removal, and dataset-defined attribute categories.

  • A.1. Data Curation for VQA LLM: The 167k VQA data includes a GQA component targeting questions whose annotated objects are critical for answering correctly.The curation first evaluates InstructBLIP on GQA questions containing annotated objects.
  • A.1. Data Curation for VQA LLM: GQA questions are retained only when InstructBLIP answers them correctly before object removal.The corresponding mentioned objects are then erased from images with LaMa for re-evaluation.
  • A.1. Data Curation for VQA LLM: The VAW object-attribution component creates open-ended and binary questions about object attributes.The attribute definitions follow the VAW dataset.
  • A.1. Data Curation for VQA LLM: Open-ended VAW questions cover color, material, hair color, pattern, face expression, pose, activity, opaqeness, and texture.These attribute types are taken from the VAW dataset.
  • A.1. Data Curation for VQA LLM: Binary VAW questions additionally cover state and optical property attributes.The binary questions include the open-ended attribute types plus these two additional types.
  • A.1. Data Curation for VQA LLM: An additional 40K examples are created from LLaVA-80K by matching question noun phrases to COCO object categories and checking for annotated instances in each image.Some COCO category names are augmented with common synonyms, including ‘man’ and ‘woman’ for ‘person’.

A.2. Data Curation for Visual Search Model

The visual search model is trained on detection, segmentation, and VQA data, including possible-location questions and LLaVA-80K instruction tuning data. Possible locations are generated for absent objects using visually similar images, captions, existing-object lists, and GPT-3.5.

  • Data sources: Training combines detection and segmentation data with VQA data containing possible-location questions and LLaVA-80K instruction tuning data.The detection and segmentation sources include COCO-Stuff, LVIS-PACO, refCOCO(+/g), and refCLEF; Objects365 v2 and GoldG provide detection data only.
  • Possible-location VQA: Possible-location supervision uses image-question-answer pairs asking where target objects are most likely located.The model’s textual contextual cues are expressed as possible-location descriptions for target objects.
  • Possible-location VQA: For each sampled COCO2017 image, two absent objects are selected from the five most CLIP-similar images.The selected objects are absent from the sampled image but appear in its five most similar images based on CLIP embeddings.
  • Possible-location VQA: GPT-3.5 predicts absent-object locations from five captions and a list of existing objects, providing the answer targets.The question template asks, “What is the most likely position of [object]”.

A.3. Model Training · A.4. Visual Search Process

The model is trained through staged feature alignment and instruction tuning, while the visual search process uses contextual cues, confidence thresholds, and heatmaps to locate target objects. The architecture combines dedicated VQA and visual-search components with different training objectives and inference procedures.

  • A.3. Model Training: The VQA LLM uses Vicuna-7b-1.3 and is trained in feature alignment and instruction tuning stages.Feature alignment trains projection modules with frozen language and vision components on a 558K LAION-CC-SBU image-text subset.
  • A.3. Model Training: The alignment stage uses batch size 256, training the linear projection for 1 epoch at learning rate 10−3 and the resampler for 5 epochs at 2 × 10−4.Instruction tuning uses 3 epochs at learning rate 2×10−5 and batch size 128.
  • A.3. Model Training: To reduce computation, the model projects the search target directly only for single-object inputs; otherwise, it projects global image features and uses the resampler for searched objects.The VWM input sequence combines image features, named target objects with coordinates, and the question; object tokens are stored in the VWM.
  • A.3. Model Training: The visual search model adopts LLaVA-7B-v1.1 with OWL-ViT-B-16, training Dcl with binary cross-entropy and DICE losses and Dtl with DETR-like set prediction and focal loss.The complete model is trained for 100K steps with batch size 64; Dcl logits provide the inference-time search cue heatmap.
  • A.3. Model Training: During training, the pre-trained MLLM uses LoRA with trainable word embeddings, while the localization image encoder and Dtl coordinates MLP remain frozen.The confidence score MLP and Dcl are trainable.
  • A.4. Visual Search Process: Visual search first applies a relatively high confidence threshold and terminates when a target exceeds it.If no target is found, the threshold is lowered and the highest-confidence candidate is accepted only if it passes the adjusted threshold.
  • A.4. Visual Search Process: For contextual search cues, the system extracts noun phrases that may express locations and prompts the MLLM to generate the corresponding search-cue heatmap.The heatmap guides the MLLM toward the visual region associated with the contextual phrase.

B. V∗Bench Examples and Two Special Subsets

V∗Bench includes examples covering attribute recognition and spatial relationship reasoning, plus two special subsets for exploratory study. One special subset, OCR, contains 30 VQA samples requiring recognition and understanding of textual characters or digits on image objects.

  • Special Subsets: The benchmark adds two special subsets for exploratory study beyond its regular tasks.The passage introduces the two subsets as an extension of the regular attribute-recognition and spatial-reasoning tasks.
  • Special Subsets: 30 VQA samples comprise the OCR subset, requiring recognition and understanding of textual characters or digits on image objects.The OCR subset targets text or digit recognition on particular objects in images.
  • Benchmark Examples: V∗Bench examples cover attribute recognition and spatial relationship reasoning tasks.Figure 8 places attribute recognition examples in the top row and spatial relationship reasoning examples in the bottom row.

C. Learning Spatial Relationship from Coordinates

V* supplies searched-target coordinates as spatial information, but the VQA LLM does not trivially understand numerical coordinates for relative spatial reasoning. Training on 46K spatial-relationship VQA examples produces a sudden loss collapse, indicating learned coordinate comparison, while mixed training may require more such data.

  • Coordinate-Based Spatial Reasoning: V* provides numerical coordinates of searched targets to the VQA LLM as spatial information for determining relative relationships.The method uses coordinates to represent the spatial information of search targets.
  • Coordinate-Based Spatial Reasoning: Training the VQA LLM only on 46K spatial-relationship VQA examples reveals a loss curve with grokking after a certain number of optimization steps.The training setup uses the constructed 46K spatial relationship related VQA data.
  • Coordinate-Based Spatial Reasoning: The loss suddenly drops to 0 after a certain number of optimization steps, suggesting the model learned to compare numerical coordinates for spatial relationships.Because grokking requires sufficient optimization steps, the spatial-relationship data ratio may need improvement when mixed with larger general multimodal instruction-tuning data.
Loading 2312.14135v2…