Source-linked AI summary
DyFo: A Training-Free Dynamic Focus Visual Search for Enhancing LMMs in Fine-Grained Visual Understanding
Geng Li, Jinglin Xu, Yunzhen Zhao, Yuxin Peng
TL;DR
Fine-grained visual tasks challenge LMMs because resolution limits and irrelevant content can obscure local details and contribute to hallucinations. DyFo uses training-free, bidirectional collaboration with visual experts and MCTS to dynamically focus image regions, improving fine-grained understanding and reducing hallucinations across fixed- and dynamic-resolution models.
Problem
LMMs struggle with fine-grained visual tasks because resolution constraints and irrelevant content hinder local-detail capture and contribute to hallucination.
Method
DyFo uses training-free bidirectional interaction between LMMs and visual experts, with MCTS treating focus regions and text as nodes for human-like visual search.
Results
DyFo improves fine-grained visual understanding and reduces visual hallucinations across fixed- and dynamic-resolution models and multiple datasets.
Takeaways & Limitations
DyFo provides a training-free dynamic-focus approach for selectively amplifying critical visual information while filtering irrelevant input in complex visual tasks.
Abstract
from arXiv · showhide
Humans can effortlessly locate desired objects in cluttered environments, relying on a cognitive mechanism known as visual search to efficiently filter out irrelevant information and focus on task-related regions. Inspired by this process, we propose Dyfo (Dynamic Focus), a training-free dynamic focusing visual search method that enhances fine-grained visual understanding in large multimodal models (LMMs). Unlike existing approaches which require additional modules or data collection, Dyfo leverages a bidirectional interaction between LMMs and visual experts, using a Monte Carlo Tree Search (MCTS) algorithm to simulate human-like focus adjustments. This enables LMMs to focus on key visual regions while filtering out irrelevant content, without introducing additional training caused by vocabulary expansion or the integration of specialized localization modules. Experimental results demonstrate that Dyfo significantly improves fine-grained visual understanding and reduces hallucination issues in LMMs, achieving superior performance across both fixed and dynamic resolution models. The code is available at https://github.com/PKU-ICST-MIPL/DyFo_CVPR2025
1. Introduction
Fine-grained visual understanding is difficult when LMMs lose small-object details or are distracted by irrelevant content. DyFo addresses this challenge with training-free, human-inspired dynamic focusing that collaborates with visual experts.
- Motivation: Visual search filters irrelevant information and concentrates processing on task-related regions, inspiring semantic selection of visual inputs.Human reading uses fixations on relevant text and saccades over irrelevant areas.
- Limitations of existing LMMs: Fixed-resolution encoders can lose small-object details after image downsizing and token limits, contributing to hallucinations on high-resolution tasks.The passage contrasts fixed-resolution processing with approaches designed for high-resolution images.
- Related approaches: Existing region-guided methods may require spatial prompts at inference, while SEAL adds visual memory and training for multi-turn visual search.These requirements limit direct applicability in real-world settings or increase adaptation costs.
- DyFo: DyFo is a training-free visual search method using bidirectional LMM–visual-expert interaction and MCTS to simulate human-like focus adjustments.It selectively amplifies critical visual information and filters irrelevant input without vocabulary expansion or specialized localization dependencies.
- Contributions: DyFo improves fine-grained visual understanding for both fixed- and dynamic-resolution models and is evaluated on hallucination and fine-grained understanding tasks.The introduction describes validation across general hallucination and fine-grained visual understanding tasks.
2. Related work
Related work positions LMMs as systems that align visual and language modalities, while DyFo is presented as a dynamic-focus framework combining visual search with multimodal models. Its framework is described as integrating Focus Adjuster and Focus Tree Search.
- Large Multimodal Models: LMMs evolved from systems dispatching tasks between language models and vision experts toward learned alignment of visual and language modalities.Examples include Visual ChatGPT, HuggingGPT, MM-REACT, and image-caption or visual question-answering training.
- DyFo framework: Figure 2 identifies DyFo’s framework components as Focus Adjuster and Focus Tree Search.The figure caption locates this composition in Section 3.2.
- Visual search with LMMs: SEAL combines localization modules and visual memory with LMMs, but requires additional training for effective visual search.DyFo is contrasted with this dependency in the supplied related-work passage.
3. Method
DyFo performs training-free visual search by coupling LMM textual focus with visual-expert region selection, then uses MCTS to navigate focus states and aggregate predictions. Its Focus Adjuster iteratively updates semantic cues and image regions, while Focus Tree Search balances exploration, exploitation, and consistency-based rewards.
- 3.1. Visual Search: DyFo frames visual search as finding an image region that maximizes the LMM’s probability for the correct answer token.The search seeks a region I such that logitθ(ŷ_t | I, X, y_<t) is as high as possible, without requiring an additional localization module or grounding-data finetuning.
- 3.2.1 Focus Adjuster: The Focus Adjuster uses bidirectional interaction: LMM text guides visual-region retrieval, and visual output redirects the LMM’s textual focus.This collaboration combines LMM multimodal understanding with visual-expert precision for fine-grained search.
- 3.2.1 Focus Adjuster: Each focus state f = (I, T) pairs a relevant image region with its primary semantic cue, and action A_i guides iterative updates.The LMM computes T^(i+1) from the current focus and action, while the visual expert computes I^(i+1) from the refined cue, previous region, and action.
- 3.2.1 Focus Adjuster: The action space includes Semantic Focus for query-relevant targets and Semantic Scatter for enlarging overly precise focused areas.These actions are designed to emulate human visual behavior rather than generate purely semantic rationales.
- 3.2.2 Focus Tree Search: Focus Tree Search builds an MCTS tree whose nodes are focus states and whose edges are focus-shift actions, balancing exploration and exploitation with UCT.Selection uses Q-values and an uncertainty term based on visit counts; expansion samples unexplored actions, and backpropagation aggregates child-node rewards into Q-values.
- 3.2.2 Focus Tree Search: The search rewards semantic consistency between image and text together with the node’s effective-area ratio, then produces the final answer through reward-weighted voting across focus-tree nodes.This voting preserves relevant detail while reducing overemphasis on details that could discard global hints.
4. Experiments
Experiments evaluate DyFo on hallucination and fine-grained understanding benchmarks, comparing fixed- and variable-resolution LMMs with existing methods and search designs. DyFo consistently improves benchmark performance, surpasses SEAL on V* Bench, and benefits from combined actions, visual-expert collaboration, and MCTS search.
- Benchmarks: DyFo is evaluated on POPE across COCO, AOKVQA, and GQA using random, popular, and adversarial sampling strategies.POPE measures visual hallucination through object-presence queries, while V* Bench evaluates fine-grained recognition and spatial reasoning in high-resolution images.
- POPE Results: DyFo consistently improves fixed-resolution and dynamic-resolution models across all POPE datasets and sampling settings.The authors attribute these gains to emphasizing relevant visual content and minimizing irrelevant factors.
- V* Bench Results: DyFo surpasses SEAL on V* Bench without additional LMM adaptation or training, despite V* Bench’s high-resolution images and small object targets.Most LMMs and tool-based methods perform near random levels on this challenging evaluation.
- Action Space: Combining two focus actions produces measurable gains, whereas using only single actions causes performance declines.This ablation evaluates how action-space design affects fine-grained visual localization on V* Bench.
- Visual Expert: DyFo consistently surpasses a standalone visual expert on POPE-COCO, indicating benefits from combining LMM reasoning with visual-expert capabilities.The authors further suggest that the combination helps mitigate biases and limitations in the visual expert.
- Efficiency: DyFo’s search space requires fewer steps than SEAL’s uniform division policy, while MCTS achieves the highest search efficiency among the compared algorithms.The search uses a fixed maximum depth of 5; the authors associate MCTS’s efficiency with reward-function modeling that avoids greedy traps.
- Case Studies: Case studies show dynamic focusing helps retain small or obscured target objects while filtering distracting visual content in low- and high-resolution images.In one high-resolution example, target objects occupy less than 1/50 of the image area, leaving nearly 98% irrelevant or distracting input.
5. Conclusion
DyFo enhances fine-grained visual understanding by selectively amplifying critical visual information and filtering irrelevant input. Its evaluations include comparisons with LLaVA-v1.5 and Qwen2-VL on POPE and V* Bench cases, while reported results indicate reduced visual hallucinations and improved performance.
- DyFo is presented as a dynamic focus visual search method for enhancing fine-grained visual understanding in LMMs.
- Figures compare DyFo with LLaVA-v1.5 and Qwen2-VL on POPE and V* Bench cases using highlighted final focus regions.
- Experimental results show that DyFo reduces visual hallucinations and enhances fine-grained visual understanding across various datasets.