Visual Agents as Fast and Slow Thinkers

• The authors argue that human-like traits in AI do not amount to System 1/2 thinking, while real-world visual queries can elicit inaccurate, overly confident responses.Source: §1 · p. 2
• A switch adapter uses System 1 for simple tasks and activates System 2 for uncertainty or small, unseen objects; proposal and segmentation adapters add region and pixel-level detail.Negative contextual data adjusts confidence and elicits relevant context before the proposal adapter outlines regions.Source: §1 · p. 2
• FAST’s hierarchical analysis moves from image-level cues to candidate boxes and pixel-level targets, while intermediate boxes and masks make its decisions directly inspectable.Source: §1 · p. 2
• FAST reports a 2.5% TextVQA accuracy gain and 6.7-point MME score improvement over LLaVA-v1.5, plus ReasonSeg gains of 3.2% CIoU and 2.7% GIoU over LISA-7B.Source: §1 · p. 2

2 METHODS

• The methods section defines System 1 and System 2 as distinct thinking modes and describes a visual agent that maps an image and prompt to a textual response.System 1 is characterized as fast and intuitive, while System 2 is deliberate and analytical.Source: §2 · p. 3

2.1 FAST

• The switch adapter is fine-tuned with positive fast-thinking data and negative examples from a 100,000-triple dataset designed around absent or visually small target objects.These examples train the adapter to identify target regions or objects needed to answer questions.Source: §2.1 · p. 3
• A switch adapter routes straightforward queries to fast responses and activates slow processing when information is missing or the target is uncertain or too small to perceive.The slow-mode training data includes missing-object cases and context clues, with 20 × 20 pixels given as the perception threshold.Source: §2.1 · pp. 3–4

2.2 PRELIMINARY

• In slow mode, FaST uses question and context clues to narrow the search to a region, then obtains a target box and, when needed, a pixel-level mask.The proposal adapter is fine-tuned to find regions from clues or object names, while the mask decoder produces target masks from the embeddings.Source: §2.2 · pp. 4–5
• FaST’s chain of evidence combines progressively deeper visual-agent steps, and its region boxes and object masks provide intermediate symbols for direct inspection.After obtaining a proposal and mask, the framework combines them with the image and question to produce a reasoning answer.Source: §2.2 · p. 5

2.3 IMPLEMENTATION DETAILS

• FaST uses a visual agent built on LLaVA-v1.5, with a CLIP-ViT-L-336px visual encoder that resizes or pads images to 336 × 336 pixels.The mask decoder follows SAM and is fine-tuned on semantic and referring segmentation datasets; a cross-attention visual sampler reduces image tokens from 256 to 32 for the chain of evidence.Source: §2.3 · p. 5

3.1.1 EXPERIMENTS ON VQA AND MULTIMODAL BENCHMARKS

• FAST exceeds LLaVA-v1.5 by 2.3% on VQAv2 and 1.8% on GQA, with gains of 6.7 on MME, 1.5 on SEED, and 0.5 on MM-Vet.Table 1 covers eight VQA and multimodal benchmarks; the comparison uses the same basic CLIP visual encoder.Source: §3.1.1 · pp. 6–7, Table 1 · p. 7

3.1.2 EXPERIMENTS ON REFERRING AND REASONING SEGMENTATION

• FAST gains 3.2% in CIoU and 2.7% in GIoU over LISA on referring and reasoning segmentation benchmarks.Table 2 reports competitive referring-segmentation results and superior reasoning-segmentation performance, particularly against LISA-7B.Source: Table 2 · p. 7, §3.1.2 · p. 8

3.2 ANALYSIS OF SYSTEM SWITCHING ADAPTER

• FAST runs 31% faster than System 2 Only on MME and 50% faster on GQA, with comparable results.System 2-mode accuracy rates are 52.2% for MME and 56.8% for GQA, combining System 1 for simpler query components with System 2 for complex aspects.Source: §3.2 · p. 8, Table 3 · p. 8, Figure 5 · p. 8

3.3 ABLATION STUDY

• Adding the proposal adapter, segmentation adapter, or both progressively improves performance; VQAv2 accuracy rises from 62.1% to 63.8%.The baseline excludes both adapters and relies on the switch adapter.Source: §3.3 · p. 8, Table 4 · p. 8
• Including both missing objects and context clues achieves the best performance on metrics including MME and refCOCOg.Table 5 analyzes the switch adapter's role in incorporating these features.Source: §3.3 · pp. 8–9, Table 5 · p. 8

3.4 QUALITATIVE COMPARISONS OF FAST

• Qualitative examples compare FAST with LLaVA-v1.5 on VQA and LISA-7B on segmentation, highlighting FAST's focus on relevant objects and pixel-level masks.The examples include smaller or less obvious segmentation targets.Source: §3.4 · p. 9, Figure 6 · p. 9

4 RELATED WORKS

• End-to-end visual agents use curated visual instruction data for question answering and tailored downstream tasks, while tool-using agents select pretrained visual perception tools.Tool outputs can improve interpretability, but their complexity may introduce inaccuracies and abundant information may obscure query-relevant details.Source: §4 · pp. 9–10
• End-to-end instruction-tuned agents are characterized as System 1-style and may give direct answers prone to hallucination.The cited discussion also notes that System 2-style reasoning can hallucinate, while increasingly complex trees and graphs can limit general applicability and propagate errors; feedback and iterative refinement are proposed as mitigations.Source: §4 · pp. 9–10
• FAST assesses information sufficiency and, when slow thinking is activated, uses multiple experts to construct a coherent chain of evidence.The authors present this approach as supporting accurate, interpretable responses and the reliability and transparency of visual agents.Source: §4 · p. 10
• FAST introduces a chain of evidence that enriches multimodal reasoning steps with cascading image-based information.The authors describe this as mirroring human visual and System 2 cognitive processes and improving accuracy, interpretability, and generalization.Source: §4 · p. 10

5 CONCLUSION

• FAST combines fast, intuitive System 1 and slow, deliberate System 2 thinking through a flexible switch that adapts to query complexity.Simple tasks receive quick responses, while more complex scenarios use hierarchical reasoning.Source: §5 · p. 10
• FAST uses a neuro-symbolic decision-making pipeline with interpretable intermediate outputs.The conclusion identifies these outputs as enabling explainability.Source: §5 · p. 10
• FAST improves results across benchmarks, and ablations highlight contextual clues, symbolic reasoning, and pixel-level adapters as important components.The conclusion links the chain of evidence with reducing hallucinations and improving interpretability.Source: §5 · p. 10

7 ETHICAL SAFEGUARDS

• The release protocols cover both models and datasets and include usage guidelines, access controls, safety filters, and monitoring systems.The authors frame these measures as intended to support responsible application.Source: §7 · p. 11
• The authors state that the safeguards aim to protect the rights and privacy of stakeholders.They describe this as part of promoting responsible and ethical research.Source: §7 · p. 11
• The authors present the safeguards as measures to prevent potential misuse and promote responsible application.The stated measures accompany the final release of models and datasets.Source: §7 · p. 11

8 REPRODUCIBILITY

• The experiments used eight NVIDIA A100-80GB SXM GPUs, and reproducing finetuning requires approximately 15 A100 GPU days.The implementation is in PyTorch, and the authors state that the full implementation would be released upon paper acceptance.Source: §8 · p. 11

SUMMARY OF THE APPENDIX

• The appendix lists further implementation details and reports additional results for different thinking modes, visual question answering, and segmentation.It also includes an analysis of failure cases and limitations and future work.Source: SUMMARY OF THE APPENDIX · p. 19
• The appendix also lists data licenses for comparison methods and sections on social impact and dataset ethical safeguards.These are identified as topics in the appendix overview.Source: SUMMARY OF THE APPENDIX · p. 19

A IMPLEMENTATION DETAILS AND PSEUDO-CODE OF FAST

• A visual resampler compresses encoder features into fixed-size latent tokens through repeated cross-attention and feedforward layers.The latent embeddings query visual features and are refined across layers.Source: A IMPLEMENTATION DETAILS AND… · p. 19
• Fine-tuning uses LLaVA-v1.5, 336 × 336 image inputs, AdamW, DeepSpeed ZeRO 2, and LoRA with rank 128 and alpha 256.The vision-encoder projection learning rate is 2e-5, and an additional resampler projection reduces image tokens.Source: A IMPLEMENTATION DETAILS AND… · p. 19
• Switch-adapter fine-tuning combines LLaVA-v1.5 instruction data with examples designed to require recognition of missing or essential objects.The pipeline includes negative data and curated GQA and VAW questions filtered by object removal and re-evaluation.Source: A IMPLEMENTATION DETAILS AND… · pp. 19–20
• LLaVA-80K noun phrases are matched to COCO categories, and instances with bounding-box annotations become target objects for training.Common synonyms such as “man” and “woman” are also matched to the person category.Source: A IMPLEMENTATION DETAILS AND… · p. 20
• GPT-4 and GPT-4V expand segmentation-related question data, with GPT-4V generating image-tailored question-answer pairs from captions, object names, and bounding boxes.The earlier caption-based approach had limited diversity; the refined pipeline uses dynamically crafted prompts to select instances.Source: A IMPLEMENTATION DETAILS AND… · p. 20
• Proposal training uses LRP++ to construct region targets and samples one question per image from 665k LLaVA instruction-tuning examples.Questions and context clues are formatted to elicit a bounding box; generated questions are expected to relate at least two objects.Source: A IMPLEMENTATION DETAILS AND… · pp. 20–21
• The segmentation adapter trains on semantic, referring, and reasoning segmentation datasets, while referring and reasoning datasets are excluded from benchmark evaluation to prevent data leakage.The semantic datasets include ADE20K, COCO-Stuff, and LVIS-PACO; ReasonSeg supplies reasoning-segmentation training data.Source: A IMPLEMENTATION DETAILS AND… · p. 21

B MORE RESULTS FOR DIFFERENT THINKING MODES

• FaST achieves the highest accuracy on challenging System 2 tasks across GQA, VQAT, and SQAI, while remaining competitive on simpler System 1 tasks.Across VQA datasets, it outperforms baselines in most cases.Source: B MORE RESULTS FOR DIFFERENT THINKING… · p. 21
• 48.2 CIoU on ReasonSeg System 2 exceeds LISA-7B at 43.3 CIoU and LLaVA at 42.4 CIoU.The passage reports significant System 2 improvements over both baselines.Source: B MORE RESULTS FOR DIFFERENT THINKING… · p. 21
• The reported results support FaST’s balance of performance across simple and complex reasoning tasks and its robustness across the evaluated task families.The authors attribute dynamic task allocation to the switch adapter.Source: B MORE RESULTS FOR DIFFERENT THINKING… · p. 21

C MORE QUALITATIVE RESULTS FOR VISUAL QUESTION ANSWERING

• In Figure 7, FaST uses world knowledge to identify a keyboard, then locate a hidden mouse and answer correctly.The example is described as integrating contextual information.Source: C MORE QUALITATIVE RESULTS FOR VISUAL… · p. 21
• Table 6 compares System 1 and System 2 performance on VQA datasets, and its caption states that FaST outperforms baselines in both modes.Source: Table 6 · p. 22
• The authors describe the qualitative results as evidence of FaST’s robustness and versatility across diverse VQA tasks.The passage characterizes the examples as challenging scenarios.Source: C MORE QUALITATIVE RESULTS FOR VISUAL… · p. 22

D MORE QUALITATIVE RESULTS FOR SEGMENTATION

• Figure 8 shows FaST identifying a wall-mounted appliance from contextual clues in a query about what can be turned on when feeling hot.The authors also describe accurate localization of difficult targets and complex reasoning on demanding queries.Source: D MORE QUALITATIVE RESULTS FOR… · p. 22
• Table 7 covers System 1 and System 2 performance on reasoning segmentation tasks; its caption describes strong performance across both tasks.Source: Table 7 · p. 23

E FAILURE CASE

• Failure cases include missed System 2 triggers, inadequate contextual clues, inaccurate region proposals, and imprecise masks, especially for small or occluded objects.The authors identify these as patterns behind suboptimal outputs.Source: E FAILURE CASE · p. 23

F LIMITATION AND FUTURE WORK

• FaST's switch adapter may miss real-world complexity because it relies on predefined negative training data, while its mask decoder may struggle with highly textured images.The authors also call for further validation across domains and tasks and plan to improve generalization and reduce real-time computational overhead.Source: F LIMITATION AND FUTURE WORK · p. 25
• The authors contrast FaST's transparent multimodal reasoning and dynamic efficiency–accuracy balance with the computational demands of large reasoning models such as OpenAI o1.They describe FaST as complementary to scalable reasoning approaches.Source: F LIMITATION AND FUTURE WORK · p. 25

G SOCIAL IMPACTS

• The paper identifies potential bias and discrimination from reliance on large models and extensive datasets, and calls for responsible deployment guidelines.It cites vision-based dialogue and security surveillance as possible applications.Source: G SOCIAL IMPACTS · p. 25

H ETHICAL SAFEGUARDS

• The authors say release protocols include usage guidelines, access controls, safety filters, and monitoring systems to promote responsible research.They also state goals of protecting stakeholder rights and privacy.Source: H ETHICAL SAFEGUARDS · p. 25

I REPRODUCIBILITY

• Experiments used eight NVIDIA A100-80GB GPUs, and reproducing fine-tuning would require approximately 15 A100 GPU days.The implementation is in PyTorch, and the review version is available at an anonymous link.Source: I REPRODUCIBILITY · pp. 25–26

J LICENSES FOR EXISTING ASSETS

• The comparison methods are publicly available for academic use, and the switch, proposal, and segmentation adapters use released code under MIT or Apache-2.0 licenses.The switch adapter uses vstar code under MIT; the proposal and segmentation adapters use Chain-of-Spot and LISA code under Apache-2.0.Source: J LICENSES FOR EXISTING ASSETS · p. 26