PaperlayerView exact revision on arXiv

Source-linked AI summary

VLM-FO1: Bridging the Gap Between High-Level Reasoning and Fine-Grained Perception in VLMs

Peng Liu, Haozhan Shen, Chunxin Fang, Zhicheng Sun, Jiajia Liao, Tiancheng Zhao

arXiv:2509.25916v1cs.CVcs.CL

TL;DR

VLM-FO1 addresses VLMs’ difficulty with fine-grained localization by retrieving features from proposed regions rather than generating coordinates. Its region encoder and token-based references integrate with pre-trained VLMs, and the paper reports state-of-the-art results across diverse perception benchmarks.

  • Problem

    VLMs struggle with fine-grained perception requiring precise localization, while generating exact coordinates is an unnatural and brittle task for language-centric models.

    Source: §1 · p. 1

  • Method

    VLM-FO1 retrieves features from proposed regions as region tokens, using a dual-vision HFRE and token-based references to integrate them with a pre-trained VLM.

    Source: §1 · pp. 2–3, §3.1 · p. 5, §3.2 · p. 5

  • Results

    VLM-FO1 reports state-of-the-art results across object grounding, region-level understanding, and visual reasoning; its lightweight 3B model achieves 44.4 mAP on COCO.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    The paper presents VLM-FO1 as a plug-and-play approach for adding fine-grained perception to pre-trained VLMs while preserving their original capabilities.

    Source: §1 · p. 3

Abstract

from arXiv · show

Vision-Language Models (VLMs) excel at high-level scene understanding but falter on fine-grained perception tasks requiring precise localization. This failure stems from a fundamental mismatch, as generating exact numerical coordinates is a challenging task for language-centric architectures. In this paper, we introduce VLM-FO1, a novel framework that overcomes this limitation by reframing object-centric perception from a brittle coordinate generation problem into a robust feature retrieval task. Our method operates as a plug-and-play module that integrates with any pre-trained VLM. It leverages a Hybrid Fine-grained Region Encoder (HFRE), featuring a dual vision encoder, to generate powerful region tokens rich in both semantic and spatial detail. A token-based referencing system then enables the LLM to seamlessly reason about and ground language in these specific visual regions. Experiments show that VLM-FO1 achieves state-of-the-art performance across a diverse suite of benchmarks, demonstrating exceptional capabilities in object grounding, region generational understanding, and visual region reasoning. Crucially, our two-stage training strategy ensures that these perception gains are achieved without compromising the base model's general visual understanding capabilities. VLM-FO1 establishes an effective and flexible paradigm for building perception-aware VLMs, bridging the gap between high-level reasoning and fine-grained visual grounding.

1. Introduction

VLM-FO1 addresses VLMs’ difficulty with precise visual localization by retrieving features from proposed regions rather than generating coordinates. It combines region tokens with pre-trained VLMs and reports strong results across perception benchmarks.

1. Introduction

  • VLMs struggle with fine-grained tasks requiring precise localization, limiting their use in settings where understanding an object’s location matters.The paper contrasts specialized detectors’ 50–60 mAP on COCO with a recall below 40% for Qwen2.5-VL-72B.Source: §1 · p. 1
  • Generating exact coordinate strings is brittle for language models: a single incorrect token can invalidate a box, and multiple instances lengthen the sequence.The paper attributes this difficulty to an architectural mismatch with sequential language generation.Source: §1 · p. 1
  • Prior approaches quantize coordinates, add prediction heads, or use region proposals, but the latter can require joint detector training or training an architecture from scratch.The paper says these proposal-based approaches fail to use pre-trained VLMs’ visual understanding and world knowledge.Source: §1 · pp. 1–2
  • VLM-FO1 treats bounding boxes as visual prompts, retrieves their features as region tokens, and adds a plug-and-play HFRE with dual vision encoders.Its two-stage, decoupled design separates proposal-model and VLM training and permits pairing with different proposal detectors.Source: §1 · p. 3
  • VLM-FO1 reports state-of-the-art results across perception benchmarks and achieves 44.4 mAP on COCO, over 20 points better and on par with specialized detectors.The introduction also reports strong results on referring expression comprehension, counting, and OCR.Source: §1 · p. 3

2. Related Work

Related work covers VLM architectures, methods that add detection capabilities, and visual-prompt approaches. VLM-FO1 is distinguished by its plug-and-play region encoder and ability to reject prompted categories absent from an image.

2.1. Vision-Language Models (VLMs)

  • VLMs typically combine visual tokens from a vision encoder with text tokens in an LLM backbone, supporting visual dialogue and reasoning.The section names LLaVA, QwenVL, and InternVL as widely adopted open-source families.Source: §2.1 · p. 3

2.2. VLMs with Detection Enhancement

  • Detection-enhanced VLMs use detection heads, visual experts, or autoregressive coordinate generation; VLM-FO1 instead adds a plug-and-play HFRE and can reject prompted categories absent from an image.The paper contrasts this design with prior region-token retrieval methods that may require base-model modifications or joint detector training.Source: §2.2 · pp. 3–4

2.3. VLMs with Visual Prompt

  • Visual prompts include bounding boxes, symbolic markers, pixel-level features or annotations, and learnable pixel-space perturbations.The cited examples include free-form markers such as scribbles and arrows.Source: §2.3 · p. 4

3. Methodology

VLM-FO1 augments a pre-trained VLM with proposal-region features and addressable region tokens, then trains the added perception components in two stages. Its encoder combines semantic context with high-resolution visual detail.

3.1. Model Architecture

  • The dual-vision encoder combines the VLM’s semantic primary features with high-resolution auxiliary features, fuses multi-scale proposal-region features, adds positional information, and projects them into region tokens.The auxiliary encoder supplies fine-grained cues, while the primary encoder provides global image tokens and semantic context.Source: §3.1 · p. 5

3.2. Grounding Language to Regions via Token-based Referencing

  • Each region feature is paired with a unique index token, allowing prompts to refer to specific regions and outputs to associate grounded phrases with region indices.The output format uses <ground> and <object> tokens to mark grounded phrases and their corresponding region indices.Source: §3.2 · p. 5

3.3. Training Strategy

  • Stage 1 freezes the pre-trained VLM and trains the HFRE and connector; Stage 2 unfreezes the auxiliary encoder, HFRE, connector, and LLM while keeping the primary encoder frozen.The first stage aligns region tokens with the LLM feature space; the second uses broader perception-focused instruction data.Source: §3.3 · pp. 5–6

4. Experiments

The experimental setup uses Qwen2.5-VL with a DaViT-Large auxiliary encoder and limits each image to 100 OPN proposals. Stage 1 training draws on detection, grounding, and region-caption datasets.

4.1. Experimental Setup

  • Experiments use Qwen2.5-VL, a DaViT-Large auxiliary encoder, and at most 100 OPN proposals per image.The reported hybrid region feature has dimension 5888; learning rates are 1e-3 for Stage 1 and 1e-5 for Stage 2.Source: §4.1 · p. 6
  • Stage 1 region-language alignment uses object detection datasets COCO, O365, and V3Det, plus GOLDG grounding data and Rexverse-2M region captions.These datasets are used to align HFRE visual features with the LLM embedding space.Source: §4.1 · p. 6

Model LVIS PACO

VLM-FO1 performs strongly across object grounding, region-level understanding, and visual region reasoning, while retaining the base model’s general visual capabilities. Its second training stage combines perception tasks with general VLM instruction data.

Model LVIS PACO

  • Stage 2 adds REC, grounding, region captioning and reasoning, counting, region QA, and OCR data, plus rejection samples and general VLM instruction data.Rejection samples cover 20% of detection-related data and prompt the model to find absent objects; the general-task data is included to mitigate catastrophic forgetting.Source: Model LVIS PACO · p. 7

4.2. Main Results

  • VLM-FO1 achieves 59.0% accuracy on COCOText, over 13 points above the next-best model, and a score of 80.1 on Ferret Bench referring reasoning.It also sets state-of-the-art region classification on LVIS and PACO, outperforming significantly larger 8B and 13B models.Source: Table 3 · p. 7, Table 4 · p. 7, Table 2 · p. 7
  • On ODinW13, VLM-FO1 achieves the highest score under standard mAP evaluation; on OVDEval, which tests linguistic labels with hard negatives, it surpasses other VLMs and Grounding DINO.Source: §4.2 · p. 7
  • VLM-FO1 earns top-tier results on Refcoco/+/g and state-of-the-art performance on HumanRef, while detect-then-count results outperform GPT-4V and open-source models up to 72B parameters.For counting, it first localizes target instances and then aggregates their count.Source: Table 5 · p. 8, Table 6 · p. 8, §4.2 · pp. 8–9

4.3. Preservation of General VLM Capabilities

  • Across MMBench, AI2D, and MMStar, VLM-FO1’s average score remains negligibly different from the base Qwen2.5-VL model.The paper attributes this preservation to freezing the core VLM during initial region-language alignment and mixing general VLM instruction data into the second stage.Source: Table 7 · p. 9, §4.3 · p. 9

5. Ablation Studies

The ablations assess HFRE components and the Simple Feature Pyramid using a representative subset of the full perception SFT dataset. The reported comparisons favor combining both region-feature sources, freezing the primary encoder, and using the feature pyramid.

5. Ablation Studies

  • Ablations use a representative subset of the full perception SFT dataset, and “Average Score” averages benchmark scores reported in previous sections.The studies are intended to validate individual model components.Source: §5 · p. 9

5.1. Effectiveness of Components in HFRE

  • 67.65 average score: full VLM-FO1-3B, combining primary and auxiliary region features with frozen QwenViT, scores highest among the reported HFRE configurations.Using only auxiliary features scores 65.89, only primary features scores 66.15, and fine-tuning the primary encoder scores 66.35.Source: §5.1 · p. 9

5.2. Effectiveness of the Simple Feature Pyramid

  • 66.15 to 64.94 average score: removing SimpleFP lowers performance when using only the primary region feature.The authors identify constructing a multi-scale pyramid from the ViT’s single-scale feature map as critical for extracting high-quality region features.Source: §5.2 · pp. 9–10

6. Conclusion

VLM-FO1 reframes object-centric perception as feature retrieval and enhances pre-trained VLMs with a plug-and-play Hybrid Fine-grained Region Encoder. The paper reports strong perception results across varied benchmarks without compromising the base model’s general-purpose abilities.

6. Conclusion

  • VLM-FO1 reframes object-centric tasks from coordinate generation to feature retrieval, using a plug-and-play design and Hybrid Fine-grained Region Encoder to enhance pre-trained VLMs.The conclusion describes the framework as bridging VLM high-level reasoning and fine-grained perception.Source: §6 · p. 10
  • VLM-FO1 achieves exceptional performance across benchmarks spanning object grounding, region-level understanding, and complex visual reasoning, often outperforming much larger models.Source: §6 · p. 10
  • The reported perception gains do not compromise the base model’s original general-purpose abilities.The conclusion attributes this result to the training strategy and architectural design.Source: §6 · p. 10

7. Comprehensive Dataset Overview for VLM-FO1

The section introduces a comprehensive overview of datasets used in VLM-FO1’s two-stage training methodology.

7. Comprehensive Dataset Overview for VLM-FO1

  • The section presents datasets used in VLM-FO1’s two-stage training methodology in Table 10.Source: §7 · p. 16

8. Additional Visualizations

The supplementary visuals provide examples of VLM-FO1 across object detection, referring expression comprehension, counting, grounding, OCR, region captioning, region VQA, and region reasoning. Table 10 summarizes the datasets used in two-stage training.

8. Additional Visualizations

Loading 2509.25916v1…