PaperlayerView exact revision on arXiv

Source-linked AI summary

AVIS: Autonomous Visual Information Seeking with Large Language Model Agent

Ziniu Hu, Ahmet Iscen, Chen Sun, Kai-Wei Chang, Yizhou Sun, David A Ross, Cordelia Schmid, Alireza Fathi

arXiv:2306.08129v3cs.CVcs.AIcs.CL

TL;DR

Knowledge-intensive visual questions can challenge vision-language models and may require external tools and multiple decisions. AVIS uses human-informed, dynamic tool selection and reasoning over tool outputs; it reports state-of-the-art results on Infoseek and OK-VQA, including 50.7% accuracy on Infoseek’s unseen entity split versus PALI [6] at 16.0%.

  • Problem

    Knowledge-intensive visual questions can require external APIs and web search, while current vision-language models struggle with fine-grained details and reasoning.

    Source: §1 · pp. 1–2

  • Method

    AVIS iteratively uses an LLM planner to select tools and queries, a reasoner to extract useful information, and working memory, guided by human decision data.

    Source: §1 · p. 2, §3.1 · p. 5, §5 · p. 10

  • Results

    AVIS reports state-of-the-art results on Infoseek and OK-VQA; on Infoseek’s unseen entity split, it achieves 50.7% accuracy versus PALI [6] at 16.0%.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The reported results show that AVIS’s human-informed, dynamic tool-use framework performs well on knowledge-intensive visual question answering benchmarks.

    Source: §1 · p. 3, §5 · p. 10

    AVIS is currently designed specifically for visual question answering and depends on the computationally intensive PALM model.

    Source: §5 · p. 10

Abstract

from arXiv · show

In this paper, we propose an autonomous information seeking visual question answering framework, AVIS. Our method leverages a Large Language Model (LLM) to dynamically strategize the utilization of external tools and to investigate their outputs, thereby acquiring the indispensable knowledge needed to provide answers to the posed questions. Responding to visual questions that necessitate external knowledge, such as "What event is commemorated by the building depicted in this image?", is a complex task. This task presents a combinatorial search space that demands a sequence of actions, including invoking APIs, analyzing their responses, and making informed decisions. We conduct a user study to collect a variety of instances of human decision-making when faced with this task. This data is then used to design a system comprised of three components: an LLM-powered planner that dynamically determines which tool to use next, an LLM-powered reasoner that analyzes and extracts key information from the tool outputs, and a working memory component that retains the acquired information throughout the process. The collected user behavior serves as a guide for our system in two key ways. First, we create a transition graph by analyzing the sequence of decisions made by users. This graph delineates distinct states and confines the set of actions available at each state. Second, we use examples of user decision-making to provide our LLM-powered planner and reasoner with relevant contextual instances, enhancing their capacity to make informed decisions. We show that AVIS achieves state-of-the-art results on knowledge-intensive visual question answering benchmarks such as Infoseek and OK-VQA.

1 Introduction

Knowledge-intensive visual questions can challenge both people and vision-language models because they require fine-grained recognition, reasoning, and external information. AVIS addresses this with dynamically selected tool use informed by human decision-making data.

1 Introduction

  • Visual information-seeking questions can require APIs or web search, while current vision-language models struggle with fine-grained details, reasoning, and image comparison.The paper notes that many such questions pose a challenge even for humans.Source: §1 · p. 2
  • AVIS uses an LLM-powered planner, reasoner, and working memory to select external tools, analyze their outputs, and retain acquired information.Its tools include computer-vision tools, web search, and image search.Source: §1 · p. 2
  • Unlike approaches that plan all steps in advance, AVIS chooses each API and query using information acquired from previous calls.The paper motivates this dynamic approach because complex API outputs cannot be predicted before a call.Source: §1 · p. 3
  • A user study supplies decision examples that shape a transition graph of available actions and contextual guidance for the planner and reasoner.The graph constrains actions at each state, while examples guide API selection and query formulation.Source: §1 · p. 3
  • AVIS reports state-of-the-art results on Infoseek and OK-VQA, including 50.7% accuracy on Infoseek’s unseen entity split versus PALI [6] at 16.0%.The paper presents this as a contribution alongside its framework and use of human decision-making data.Source: §1 · p. 3

2 Related Work

Prior work integrates LLMs with tools for multimodal tasks and uses agents to adapt actions from feedback. AVIS differs by combining dynamic tool decisions with a human-derived transition graph and decision examples.

Augmenting LLMs with Tools

  • Related work includes LLM tools for external knowledge and mathematics, as well as vision systems that call foundation models or decompose tasks into programs.Examples include TALM [31], ToolFormer [35], Visual ChatGPT [40], VisProg [13], and ViperGPT [36].Source: §2 · p. 3
  • Many LLM-tool vision systems use plan-then-execute workflows, which cannot revise API calls based on tool outputs and often fail on complex tasks.Such tasks can require modifications based on real-time feedback.Source: §2 · pp. 3–4

Decision Making with LLM as an Agent

  • LLM agents can adapt to real-time feedback, but unrestricted tool choices create large search spaces and may lead to loops or propagated errors.AVIS guides decisions to mimic human behavior, using human prior information at decision time rather than retraining.Source: §2 · p. 4
  • AVIS differs from Tree-Of-Thought [43] by constraining tree search with a human-defined transition graph and guiding decisions with a dynamic prompt manager.The paper also notes that Tree-Of-Thought’s success suggests tree-search ideas may apply to other LLM reasoning tasks.Source: §2 · p. 4

3 Method

AVIS iteratively selects tools, evaluates their outputs, and updates its working memory until it can answer or needs another search. A user study informs both the actions available at each state and the contextual examples used for decisions.

3.1 General Framework

  • The planner uses the current state and working memory to filter feasible actions, retrieves relevant human examples, and prompts an LLM to choose the next tool and query.Previously taken actions are excluded to reduce the search space.Source: §3.1 · p. 5
  • After each tool call, the reasoner classifies its output as informative, uninformative, or a final answer, updating the state or returning to the planner as appropriate.Working memory retains extracted information throughout the repeated workflow.Source: §3.1 · pp. 5–6
  • AVIS’s dynamic decision-making and backtracking ground tool choices in current execution results, unlike plan-then-execute methods.The method iteratively searches for combinations of tools.Source: §3.1 · p. 6

3.2 Tools and their APIs

  • AVIS’s tools include captioning, VQA, object detection, image search, OCR, web search, and LLM question answering.Image search can return knowledge-graph entities, product titles, and captions of similar or identical images.Source: §3.2 · p. 6

3.3 Gathering User Behavior to Inform LLM Decision Making

  • The user study records users’ tool actions and received outputs while they answer visual questions using tools also available to AVIS.The study involved 10 participants answering 644 questions randomly selected from Infoseek [7] and OK-VQA [26].Source: §3.3 · pp. 6–7
  • User decisions define a transition graph that restricts actions at each state, while decision examples provide contextual guidance to the planner and reasoner.At START, the available actions are PALI caption, PALI VQA, or object detection.Source: §3.3 · p. 7

4 Experiments

AVIS is evaluated on OK-VQA and InfoseekWikidata, which require commonsense and fine-grained knowledge, respectively. It outperforms fixed-sequence baselines using the same tools and reports strong results on both datasets.

Experimental Setup

  • The evaluation covers OK-VQA’s commonsense questions and InfoseekWikidata’s finer-grained knowledge needs; AVIS uses 10 in-context examples per dataset.The experiments use frozen PALM 540B for the planner and reasoner and report VQA accuracy.Source: §4 · pp. 7–8

Experimental Results

  • On InfoseekWikidata, AVIS reaches 50.7 accuracy on the unseen entity split and 56.4 on the unseen question split, versus PALI-17B’s 16.0 and 20.7.AVIS uses no fine-tuning and is guided by 10 in-context examples.Source: Table 1 · p. 8, §4 · p. 9
  • Using the same tools, AVIS outperforms the best sequential baseline by up to 17.3 accuracy on InfoseekWikidata; removing any tool group lowers accuracy.Object and Search are more important than PALI for this dataset, according to the reported ablation.Source: §4 · p. 9
  • On OK-VQA, AVIS with all tools achieves the best performance, with 60.2 accuracy, while remaining below but comparable to fine-tuned PALI.Object detection is less crucial here than PALI captioning and VQA.Source: Table 2 · p. 9, §4 · p. 10

Case studies for dynamic decision making

  • Case studies show AVIS changing tools across stages and backtracking when the reasoner finds retrieved information irrelevant.In the fungi example, it rejects an irrelevant leaf object, replans, and identifies Stereum.Source: Figure 5 · p. 9, §4 · p. 10

5 Conclusion

AVIS uses tree-search to iteratively coordinate an LLM planner and reasoner with tools for knowledge-intensive visual question answering. The authors identify visual-question scope and reliance on computationally intensive PALM as current boundaries.

5 Conclusion

  • AVIS iteratively uses an LLM-powered planner and reasoner with tools until it gathers the information needed to answer a visual question.The planner selects tools and forms queries; the reasoner processes tool outputs and extracts key information.Source: §5 · p. 10
  • The current framework is specifically designed for visual question answering, with extension to other reasoning tasks left for future work.Source: §5 · p. 10
  • AVIS depends on computationally intensive PALM, and the authors propose investigating whether lighter language models can perform this decision-making framework.Source: §5 · p. 10

A Implementation of AVIS workflow

AVIS uses Google Search, LENS, PALI, and PALM directly in its experiments, with some tool outputs precomputed and others called at runtime.

A Implementation of AVIS workflow

B Comparison to pure Autonomous baseline without Transition Graph

The comparison removes transition-graph guidance and presents the model with all tools and task descriptions at each step. The unguided baseline performs below AVIS with the graph and user prompts.

B Comparison to pure Autonomous baseline without Transition Graph

C Analysis of AVIS’s generated tool execution sequence

AVIS’s tool traces show recurring multi-step patterns rather than uniform tool combinations, including an initial visual-information phase followed by knowledge retrieval. The reasoner marks some tool outputs as uninformative and triggers backtracking.

C Analysis of AVIS’s generated tool execution sequence

D Dataset Details

Infoseek targets questions that require information beyond common sense, while OK-VQA questions require external knowledge beyond what is visible in images. The section also identifies the reported dataset splits and accompanying tool-frequency and reasoner-example visuals.

D Dataset Details

  • Infoseek questions cannot be answered merely through common-sense knowledge, while OK-VQA requires external knowledge beyond image content.Infoseek questions were generated by integrating human-annotated questions with visual entity recognition datasets and Wikidata; OK-VQA was crowdsourced using Wikipedia knowledge resources.Source: D Dataset Details · pp. 14–15
  • Figures 9 and 10 show the frequencies of the third and fourth most commonly called tools, respectively.Source: Figure 9 · p. 15, Figure 10 · p. 15
  • Table 4 presents examples of API execution results and the reasoner’s justifications.Source: Table 4 · p. 15
  • At submission, only Infoseek’s Wikidata split was accessible, and the paper also reports results on its human split in Table 5.Source: D Dataset Details · p. 15

E Prompt Examples

This section presents AVIS prompt examples for planning, question decomposition, object selection, and answer reasoning. The examples show how prompts guide tool choice, track prior actions and context, and support concise, evidence-grounded answers to visual and knowledge questions.

E Prompt Examples

  • Planner prompts pair tool-selection instructions with user-decision examples to guide which tool to use and what query to send.Task instructions distinguish straightforward visual questions from questions needing external knowledge or more information.Source: E Prompt Examples · pp. 15–17
  • AVIS decomposes each question at the outset into a visual sub-question and a knowledge sub-question, using the visual answer as # in the latter.Examples ask first which object, animal, sport, or food is shown, then pose the related factual question about that answer.Source: E Prompt Examples · pp. 19–20
  • Object-selection prompts ask the model to explain and choose the object ID most relevant to the query, including when contextual clues only indirectly suggest a location.An example selects an object containing Monterey pine and European rabbit information as a potential location clue, despite no explicit place name.Source: E Prompt Examples · p. 20
  • Reasoning prompts direct the model to identify relevant knowledge, extract a short answer only from supplied information, and say “cannot be answered” when evidence is insufficient.The examples contrast answering a building question from object information and retrievals with abstaining when results about snails do not identify a sport.Source: E Prompt Examples · p. 21
  • In a visual-question example, the reasoner uses an object-query suggestion to answer that the insect is a Western Tiger Swallowtail.The prompt example contrasts this informative object suggestion with retrieved results about butterflies.Source: E Prompt Examples · p. 22
  • The knowledge-question reason prompt asks the model to select the most relevant entry, give a short answer using only supplied knowledge, and say “cannot be answered” when uncertain or the knowledge is irrelevant.It also uses repeated information as a relevance cue.Source: E Prompt Examples · p. 22
  • 6.54 - 6.86 is the example answer for tennis-ball diameter in centimetres, selected from a retrieved entry reporting a 6.54–6.86 cm diameter range.The example reasons from the requested unit to the range in centimetres.Source: E Prompt Examples · p. 23
  • For the Torre Picasso question, the example selects the entry reporting 26 elevators and answers 26, despite other entries specifying that 18 serve office floors.The prompt illustrates choosing a relevant knowledge entry amid multiple retrieved results.Source: E Prompt Examples · p. 24
  • The workflow offers only untraversed actions for the current state, then stores informative results as context in working memory and repeats planning, execution, and reasoning until an answer is produced.If a result is not informative, the action is marked traversed; if the reasoner supplies an answer, execution stops.Source: E Prompt Examples · pp. 24–25

F Error Analysis

The examples illustrate three error types: planning misses important details, the QA module extracts the wrong evidence, and a tool provides incorrect information.

F Error Analysis

Related papers

Loading 2306.08129v3…