PaperlayerView exact revision on arXiv

Source-linked AI summary

V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs

Penghao Wu, Saining Xie

arXiv:2312.14135v2cs.CV

TL;DR

Current multimodal LLMs lack visual search for focusing on important details in high-resolution images. V* adds LLM-guided visual search through SEAL, and the authors report improved attribute-recognition performance with visual search.

  • Problem

    Current MLLMs rely on vision encoders that may overlook details in high-resolution images, while existing benchmarks do not adequately test detailed visual grounding.

    Source: §1 · p. 2

  • Method

    SEAL integrates an LLM-guided visual search mechanism that uses language-model knowledge to locate needed visual information, alongside V*Bench for evaluating detailed grounding in high-resolution images.

    Source: §1 · p. 2

  • Results

    With visual search, the model greatly improves performance on attribute recognition, although MLLMs remain short of near-perfect human performance.

    Source: §5.1 · p. 8

  • Takeaways & Limitations

    The study concludes that visual search capabilities play a critical role in MLLMs, while its current search model is tailored primarily to natural images and common objects.

    Source: §6 · p. 9

    The recursive non-overlapping patching strategy may fail when targets lie at patch boundaries.

    Source: §3.3.3 · p. 6

Abstract

from arXiv · show

When we look around and perform complex tasks, how we see and selectively process what we see is crucial. However, the lack of this visual search mechanism in current multimodal LLMs (MLLMs) hinders their ability to focus on important visual details, especially when handling high-resolution and visually crowded images. To address this, we introduce V*, an LLM-guided visual search mechanism that employs the world knowledge in LLMs for efficient visual querying. When combined with an MLLM, this mechanism enhances collaborative reasoning, contextual understanding, and precise targeting of specific visual elements. This integration results in a new MLLM meta-architecture, named Show, sEArch, and TelL (SEAL). We further create V*Bench, a benchmark specifically designed to evaluate MLLMs in their ability to process high-resolution images and focus on visual details. Our study highlights the necessity of incorporating visual search capabilities into multimodal systems. The code is available https://github.com/penghao-wu/vstar.

1. Introduction

The introduction argues that current MLLMs can miss important details in high-resolution images and cannot proactively seek unclear information. It presents SEAL, V∗, and V∗Bench as an architecture, search mechanism, and benchmark for improving and evaluating detailed visual grounding.

1. Introduction

  • Current MLLMs rely on often-frozen vision encoders trained on low-resolution images, so they may overlook high-resolution details and cannot proactively seek missing information.The introduction gives 224×224 and 336×336 pixels as examples of encoder training resolutions.Source: §1 · p. 2
  • V∗ uses an MLLM’s world knowledge and common sense to reason about target locations and perform informed searches across images of any resolution.Its design draws on top-down feature guidance and contextual scene guidance in human visual search.Source: §1 · p. 2
  • V∗Bench is a dedicated VQA benchmark testing whether MLLMs can ground specific, easily overlooked visual details in high-resolution images.The authors say existing MLLM benchmarks do not adequately expose this limitation of current approaches.Source: §1 · p. 2

2. Related Work

The related work contrasts earlier computational visual-search approaches with V*’s LLM-guided search, and reviews end-to-end and tool-using multimodal LLMs and their limitations.

2.1. Computational Models for Visual Search

  • Earlier visual-search models use Bayesian saliency [41], scene priors, target-template similarity, or inverse reinforcement learning [55].These approaches model aspects of human visual search.Source: §2.1 · p. 2
  • Unlike gaze-mimicking models with fixed-size windows and limited target and scene information, V* hierarchically searches at any resolution using LLM common-sense knowledge.The authors connect active search to System II cognition and describe it as a visual counterpart to LLM chain-of-thought [49].Source: §2.1 · pp. 2–3

2.2. Multimodal LLMs

  • End-to-end MLLMs connect pretrained LLMs to vision encoders through projection or alignment and joint training, but visual-feature projection can lose information.Low-resolution inputs may leave crucial details poorly captured, contributing to inaccurate or hallucinated answers.Source: §2.2 · pp. 3–4
  • LLM-tool-using systems call vision experts through reasoning [31] [53] [54] [56] [62], but text-only transfer can distort or omit visual details and compound errors.These systems typically use captioning or detection models to provide textual image information before the LLM selects further tools or answers.Source: §2.2 · p. 4

3. Method

SEAL combines a VQA LLM and a visual search model through a visual working memory (VWM); its V* search uses LLM knowledge to guide localization across image regions. The described instantiation uses LLaVA-7B as its MLLM.

3.1.1 Model Structure

  • SEAL first checks whether global image features suffice; if not, it identifies missing target objects and stores the question, image, searched crops, and their coordinates in VWM for the VQA LLM.The SEAL framework is a general meta-architecture comprising a VQA LLM and a visual search model; this work instantiates it with LLaVA-7B.Source: §3.1.1 · p. 4, §3 · p. 4

3.2. Data Curation for VQA LLM

  • VQA LLM tuning uses 100k negative target-reasoning examples, 167k VQA examples, and 120k LLaVA instruction-tuning examples.The VQA set comprises 70k GQA, 51k object-attribute, and 46k spatial-relation examples; negative examples include objects absent from the image or too small for CLIP features.Source: §3.2 · p. 5

3.3.1 Problem Formulation

  • Unlike fixed-size referring-expression comprehension, visual search must handle images at any resolution and locate targets both accurately and quickly.Exhaustively patchifying images is one possible approach, but the paper describes it as too inefficient for very high-resolution images.Source: §3.3.1 · p. 5, §3.3.2 · p. 6

3.3.2 Model Structure

  • The visual search model adds a <LOC> token and uses separate target-localization and search-cue decoders to produce target coordinates with confidence scores and a cue heatmap.The MLLM supplies contextual and location-related information for the queried expression; its localization module combines visual tokens with embeddings derived from <LOC>.Source: §3.3.2 · p. 6

3.3.3 Search Algorithm

  • V* uses target-specific heatmaps when available, otherwise asks the MLLM for contextual location cues, then recursively searches priority-ranked image patches until locating the target or reaching a size threshold.Patch division adapts to image orientation; the authors note that non-overlapping patches can fail for targets on patch boundaries.Source: §3.3.3 · p. 6

4. Benchmark

V*Bench evaluates detailed visual analysis on high-resolution images through attribute recognition and spatial relationship reasoning. On the benchmark, SEAL outperforms GPT-4V and Gemini by a large margin despite using a Vicuna-7B LLM.

Benchmark

  • V*Bench uses 191 high-resolution SA-1B images, averaging 2246×1582 pixels, to evaluate MLLMs in visually complex scenarios where relevant information may be hard to find.Source: §4 · p. 7
  • The benchmark has 115 attribute-recognition samples and 76 spatial-reasoning samples, testing object attributes and relative positions between two objects.Human annotators selected and crafted the images and questions to prevent guessing without accurately grounding the relevant objects.Source: §4 · p. 7
  • SEAL outperforms leading-edge systems such as GPT-4V and Gemini by a large margin on V*Bench despite using only a Vicuna-7B LLM.The comparison is presented in Table 1.Source: Table 1 · p. 7
  • For open-source MLLM comparisons, each question has multiple-choice options: four for open-ended questions and two for binary questions, reviewed by annotators for ambiguity.Source: §4 · p. 7

5. Experiments

Experiments find that V* improves visual-detail performance through guided search, while ablations and search comparisons support the mechanism’s value. Across broader multimodal benchmarks, adding visual search largely preserves general capabilities, with some gains and limited declines.

5.1. Evaluation on V∗Bench

  • Most evaluated MLLMs perform near random guessing on V∗Bench, while adding visual search greatly improves performance, though a substantial gap to near-perfect human performance remains.V* search takes an average of 6.0 seconds per target on one A100 GPU. GPT-4V and Gemini handle some easier attribute-recognition cases, but overall performance remains unsatisfactory.Source: §5.1 · p. 8

5.2. Ablation Study

  • The LLaVA model without visual search still struggles, and replacing search with GroundingDINO or OWL-ViT detection yields significantly inferior performance to V∗.The ablation also notes practical difficulties applying off-the-shelf detectors to very high-resolution images.Source: §5.2 · p. 8, Table 2 · p. 8

5.3. Visual Search Evaluation

  • V∗ greatly reduces average search length versus random and sequential strategies, with both target and contextual search cues contributing to the improvement.Evaluation records 245 target locations and includes only samples successfully located after search.Source: §5.3 · pp. 8–9, Table 3 · p. 8
  • On COCO-Search18 validation samples requiring search, V∗ achieves efficiency similar to human fixation guidance, while focusing on a larger contextual region than human fixations.Human fixation sequences are converted into weighted heatmaps to guide the comparison.Source: Table 4 · p. 8, Figure 6 · p. 8, §5.3 · p. 9

5.4. General Multimodal Benchmarks Evaluation

  • With visual search, performance improves on MME and POPE, remains basically unchanged on MMBench and SEED-Bench, and declines slightly on MM-Vet and LLaVA-BenchW.The authors suggest smaller benchmark size, GPT4-based evaluation uncertainty, and difficulty locating diagram items as possible reasons for the latter declines.Source: §5.4 · p. 9, Table 5 · p. 9

6. Conclusion

The conclusion presents SEAL with V* for accurate visual grounding in high-resolution images and V*Bench to highlight the role of visual search in MLLMs, while noting the current scope and future work.

6. Conclusion

  • SEAL features V*, an LLM-guided visual search algorithm for accurate visual grounding in high-resolution images.Source: §6 · p. 9
  • V*Bench highlights the critical role of visual search capabilities in MLLMs.Source: §6 · p. 9
  • The current visual search model targets natural images and common objects; extending it to documents, diagrams, long-form videos, or open-world environments requires additional training and algorithm design.Source: §6 · p. 9
  • Convolution-based architectural improvements are proposed as a way to process images of any resolution more efficiently and further improve search efficiency.Source: §6 · p. 9

A. Implementation Details · A.1. Data Curation for VQA LLM

The VQA data combines GQA and VAW questions filtered by whether removing the mentioned object prevents InstructBLIP from answering correctly, with additional LLaVA-80K samples targeted using annotated COCO-category instances.

A.1. Data Curation for VQA LLM

  • GQA questions were retained only when InstructBLIP answered correctly before LaMa erased the mentioned objects and could not answer correctly afterward.This selected questions where annotated objects were important to answering correctly.Source: §A.1 · p. 13
  • VAW data uses fixed templates for open-ended and binary attribute questions, filtered with the same InstructBLIP and object-erasing strategy as GQA.Open-ended types include color, material, hair color, pattern, face expression, pose, activity, opacity, and texture; binary questions additionally include state and optical property.Source: §A.1 · p. 13
  • For the additional 40K data created from LLaVA-80K instruction-tuning data, noun phrases in questions/instructions matched to COCO object-category names were retained only if corresponding images had annotated instances, which served as training targets with bounding boxes.Source: §A.1 · p. 13

A.2. Data Curation for Visual Search Model

The visual search model is trained on detection/segmentation data and VQA data, including possible-location questions and LLaVA-80K instruction-tuning data. Its possible-location examples use absent objects and GPT-3.5-generated answers grounded in image captions and existing objects.

A.2. Data Curation for Visual Search Model

  • Training combines detection and segmentation data with VQA data comprising possible-location questions and LLaVA-80K instruction-tuning data.Source: §A.2 · p. 13
  • COCO-Stuff, LVIS-PACO, refCOCO(+/g), and refCLEF provide both detection and segmentation data, while Objects365 v2 and GoldG provide detection data only.Source: §A.2 · p. 13
  • Possible-location QA examples pair COCO2017 images with two absent objects appearing in their five most CLIP-similar images, then ask GPT-3.5 to infer locations from five captions and existing objects.The question asks, “What is the most likely position of [object]”.Source: §A.2 · p. 13

A.3. Model Training · A.4. Visual Search Process

The sections specify two-stage training for the VQA LLM and visual search model, then describe a confidence-based search process that prioritizes finding a single target. They also define how search cues and visual features are represented for the LLM.

A.3. Model Training

  • The VQA LLM uses Vicuna-7b-1.3 and is trained in feature-alignment and instruction-tuning stages, with the vision encoder frozen only during alignment.Alignment trains the linear and resampler projection modules on a subset of the 558K LAION-CC-SBU image-text pairs used in LLaVA; instruction tuning uses 387k constructed examples and trains the LLM and projection modules.Source: §A.3 · p. 13
  • Alignment uses batch size 256: the linear projection module trains for 1 epoch at 10−3, and the resampler for 5 epochs at 2 × 10−4; instruction tuning runs 3 epochs at 2×10−5 with batch size 128.Source: §A.3 · p. 13
  • To reduce computation, training projects the search target’s feature with the linear module only for a single object; otherwise it projects the global image feature and uses the resampler for searched objects.Source: §A.3 · p. 13
  • The LLM input combines image feature tokens, named target-object tokens with bounding-box locations, and the question.Target objects are represented as <Object> features stored in the VWM, with locations formatted [x1, y1, x2, y2].Source: §A.3 · p. 13
  • The search model uses LLaVA-7B-v1.1 with an OWL-ViT-B-16 vision encoder; Dcl uses binary cross-entropy and DICE losses, while Dtl uses DETR-like set prediction with focal loss for coordinate regression.It trains for 100K steps at batch size 64 and learning rate 10−4, using general detection/segmentation, referring detection/segmentation, and VQA data at a 15:8:15 sampling ratio.Source: §A.3 · pp. 13–14
  • During training, LoRA and word embeddings are trainable in the pretrained MLLM, while the localization image encoder and Dtl coordinates MLP are frozen; the confidence-score MLP and Dcl are trainable.Source: §A.3 · p. 14

A.4. Visual Search Process

  • Search halts when a target exceeds the higher confidence threshold; if none qualifies, it lowers the threshold and accepts the highest-confidence candidate, focusing on one target rather than exhaustive search.The higher and lower thresholds are 0.5 and 0.3; δ = max(3.0, 6.0 × 0.7^l), where l is the image subdivision level. Multiple targets found directly on the global image are all added to the VWM.Source: §A.4 · p. 14
  • To generate a contextual-cue heatmap, the visual search model extracts noun phrases that express possible locations and prompts the MLLM to locate each phrase.Source: §A.4 · p. 14

B. V∗Bench Examples and Two Special Subsets

V*Bench includes examples of attribute recognition and spatial relationship reasoning, plus two exploratory subsets for OCR and GPT-4V failures. The section also notes that MM-REACT’s OCR approach can falter when text must be interpreted in context or located in a crowded scene.

B. V∗Bench Examples and Two Special Subsets

  • V*Bench adds OCR (30 samples requiring recognition of object text or digits) and GPT-4V-hard (17 samples where GPT-4V fails but the visual-search model succeeds).LLaVA-1.5, MM-REACT, GPT-4V, and SEAL are evaluated on both subsets.Source: B. V∗Bench Examples and Two Special… · p. 14
  • MM-REACT and GPT-4V perform decently on OCR, but MM-REACT can fail when images contain multiple texts or questions require understanding text context or location.MM-REACT uses an external OCR model that detects image text and provides it to the LLM bottom-up; GPT-4V likely also has an OCR module.Source: B. V∗Bench Examples and Two Special… · p. 14
  • Figure 8 shows V∗Bench examples, with attribute recognition in the top row and spatial relationship reasoning in the bottom row.The correct option is marked in green.Source: Figure 8 · p. 14

C. Learning Spatial Relationship from Coordinates

VQA LLMs do not readily infer spatial relationships from numerical target coordinates, but training on spatial-relationship VQA data can teach coordinate comparison after a delayed loss drop. The authors caution that mixed multimodal training may need a higher proportion of spatial data.

C. Learning Spatial Relationship from Coordinates

  • The VQA LLM receives searched targets’ numerical coordinates as spatial information, but understanding their relative relationships is not trivial.Source: C. Learning Spatial Relationship from… · p. 14
  • When trained only on the constructed 46K spatial-relationship VQA examples, loss suddenly drops to 0 after a certain number of optimization steps, suggesting the model learned to compare numerical coordinates for spatial relationships.Source: C. Learning Spatial Relationship from… · p. 15
  • Because grokking requires sufficient optimization steps, mixed general multimodal instruction tuning may need a higher spatial-data ratio for reliable coordinate understanding.The passage gives LLaVA-1.5 training data as an example of general multimodal instruction-tuning data.Source: C. Learning Spatial Relationship from… · p. 15

Related papers

Loading 2312.14135v2…