PaperlayerView exact revision on arXiv

Source-linked AI summary

LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding

Shihao Wang, Shilong Liu, Yuanguo Kuang, Xinyu Wei, Yangzhou Liu, Zhiqi Li, Yunze Man, Guo Chen, Andrew Tao, Guilin Liu, Jan Kautz, Lei Zhang, Zhiding Yu

arXiv:2605.27365v2cs.CVcs.AIcs.LGcs.RO

TL;DR

VLM grounding and detection often serialize coupled box geometry into sequential coordinate tokens, limiting throughput and localization coherence. LocateAnything addresses this with Parallel Box Decoding and large-scale training, achieving up to a 2.5× speedup while improving localization quality across diverse tasks.

  • Problem

    Sequentially generating independent coordinate tokens serializes coupled 2D geometry, creating a practical bottleneck and underusing coordinate correlations.

  • Method

    LocateAnything uses Parallel Box Decoding to predict each bounding box or point as an atomic geometric unit in one parallel step.

  • Results

    Up to 2.5× speedup over competitive methods accompanies improved localization quality across layout grounding, long-tail detection, and GUI grounding benchmarks.

  • Takeaways & Limitations

    LocateAnything provides a practical and scalable route toward efficient, precise visual perception for latency-sensitive embodied and interactive systems.

  • Takeaways & Limitations

    The model is primarily trained with supervised fine-tuning, leaving reinforcement learning for future work to improve robustness and worst-case decoding speed.

Abstract

from arXiv · show

Vision-language models (VLMs) commonly formulate visual grounding and detection as a coordinate-token generation problem, serializing each 2D box into multiple 1D tokens that are learned and decoded largely independently. This token-by-token decoding mismatches the coupled structure of box geometry and creates a practical inference bottleneck due to strictly sequential generation. We introduce LocateAnything, a unified generative grounding and detection framework based on Parallel Box Decoding (PBD). By decoding geometric elements such as bounding boxes and points as atomic units in a single step, LocateAnything preserves intra-box geometric coherence and unlocks substantial parallelism. We show that PBD improves both decoding throughput and localization accuracy. We further develop a scalable data engine and curate LocateAnything-Data, a large-scale dataset with more than 138 million training samples, substantially increasing data diversity for high-precision localization. Extensive evaluations show that LocateAnything advances the speed-accuracy frontier, achieving significantly higher decoding throughput while improving high-IoU localization quality across diverse benchmarks. The results highlight the complementary benefits of Parallel Box Decoding and large-scale training data in enabling efficient and precise unified visual grounding and detection.

1. Introduction

LocateAnything addresses the latency and throughput limits of sequential coordinate-token decoding by introducing Parallel Box Decoding, which predicts complete boxes or points as atomic units. The framework combines box-aligned training with flexible Fast, Slow, and Hybrid inference modes to balance speed, robustness, and localization accuracy.

  • Motivation: VLM detection and grounding commonly serialize 2D geometric objects into coordinate-token sequences, forcing sequential inference.This creates higher latency, lower throughput, and underuses correlations among x1, y1, x2, and y2.
  • Parallel Box Decoding: PBD aligns multi-token prediction blocks with box structure to improve decoding throughput and localization accuracy.The approach is designed to reconcile high-throughput decoding with reliable localization.
  • Parallel Box Decoding: LocateAnything introduces Parallel Box Decoding, predicting each bounding box or point as an atomic unit in one parallel step.Training predicts the full coordinate set (x1, y1, x2, y2) together, avoiding arbitrary coordinate chunking.
  • Inference modes: Fast Mode predicts full boxes in parallel for maximum throughput, whereas Slow Mode uses sequential coordinate decoding for high-precision outputs.Fast Mode targets latency- and compute-constrained settings; Slow Mode supports precision-oriented labeling, dataset curation, and offline evaluation.
  • Inference modes: Hybrid Mode falls back to localized sequential re-decoding when parallel outputs violate format or consistency requirements.It is intended for production pipelines requiring both speed and accuracy while preserving most parallel-decoding speed gains.

2. Related Work

Prior VLM grounding and detection methods either use task-specific heads or autoregressive coordinate-token generation, while parallel-decoding research explores multiple-token, diffusion, and block-based alternatives. LocateAnything instead decodes each complete coordinate set as one atomic block, targeting both lower latency and stronger geometric coherence.

  • Visual Detection and Grounding in VLMs: Recent VLMs formulate visual grounding and detection as autoregressive token generation, but this paradigm can suffer from structural hallucinations and high latency.Earlier approaches traditionally relied on task-specific heads.
  • Parallel Decoding via MTP and Diffusion LLMs: MTP predicts multiple future tokens simultaneously, while Future Summary Prediction uses auxiliary heads to capture long-term dependencies.These methods are proposed to mitigate autoregressive latency and are often coupled with speculative decoding.
  • Parallel Decoding via MTP and Diffusion LLMs: Diffusion and hybrid semi-autoregressive language models enable bidirectional or block-parallel decoding while retaining causal dependencies and KV-caching compatibility.The cited approaches include DiffuCoder, Block Diffusion, SDLM, and Fast-dLLM v2.
  • LocateAnything’s Distinction: LocateAnything outputs each complete bounding box in one parallel step, unlike NTP’s slow fragmented decoding and MTP’s arbitrary token chunks.Its Parallel Box Decoding treats the entire coordinate set as a single atomic block, combining high throughput with structural coherence.

3. Method

LocateAnything integrates Parallel Box Decoding into a native-resolution VLM through block-based outputs, dual NTP–MTP training, and on-demand inference. Its method also introduces structured attention patterns, fallback handling for ambiguous predictions, and the large-scale LocateAnything-Data dataset.

  • Overview: The architecture combines a Moon-ViT vision encoder, Qwen2.5 language decoder, and MLP projector, while native-resolution visual tokens preserve fine-grained spatial details.Given image ℐ, the encoder extracts visual tokens Z = Encoder(ℐ) at native resolution.
  • Block-Based Output Formulation: PBD reorganizes quantized coordinates into constant-length blocks, supporting semantic, box, negative, and end functions with <null> padding for uniform shapes.Each block has length L = 6 and accommodates a bounding box plus structural tokens such as <box> and </box>.
  • Joint NTP–MTP Training: The dual-formulation objective jointly trains causal NTP and block-wise MTP, using specialized masks to isolate streams while preserving shared context and geometric intra-block interactions.The losses are jointly minimized as ℒ = ℒntp + ℒmtp.
  • On-Demand Inference: An NTP fallback addresses format irregularity and spatial ambiguity by validating syntax and spatial confidence during MTP inference.The ambiguity trigger requires the top-1 coordinate token probability to be below 0.7 and a second condition involving the max-min difference, which is truncated in the supplied passage.
  • On-Demand Inference: PBD inference offers slow NTP, fast MTP, and hybrid modes, with committed blocks stored in the key-value cache as causal context for later steps.Fast Mode discards <null> padding tokens before appending the remaining tokens to the output.
  • LocateAnything-Data: LocateAnything-Data is curated as a large-scale, multi-domain dataset for training general-purpose visual detection and grounding models.The passage states that construction details are provided in the supplementary material.

4. Experiments

Experiments show that LocateAnything improves localization accuracy across general, dense, and open-world benchmarks while substantially increasing decoding throughput. Ablations attribute these gains to box-aligned parallel decoding, joint training, and hybrid inference.

  • High-Quality Multi-Object Detection: LocateAnything improves mean F1 by +3.8% on LVIS and +1.8% on COCO versus Rex-Omni at identical model size.The model also generalizes to unseen, heavily packed object types.
  • Superior Decoding Speed: 12.7 BPS in hybrid mode is over 10× faster than Qwen3-VL at 1.1 BPS and 2.5× faster than Rex-Omni at 5.0 BPS.Throughput is measured in boxes per second on a single NVIDIA H100 GPU with batch size 1.
  • Coordinate Representation: 52.1 mean F1 is achieved by PBD in Slow Mode, exceeding Textual and Quantized representations at 49.1 and 50.1 mean F1, respectively.The ablation supports box-aligned supervision over forced token-by-token generation.
  • MTP Formulation: 16.9 BPS and 49.6 mean F1 are achieved by PBD, outperforming structure-agnostic MTP while avoiding its speed-accuracy trade-off.SDLM-B6 reaches 46.1 F1-score at 5.5 BPS, whereas PBD aligns MTP blocks with bounding-box units.
  • Decoding Mode: 13.2 BPS and 51.6 F1 are obtained by Hybrid Mode, balancing Fast Mode’s 16.9 BPS with Slow Mode’s accuracy through joint training.Joint training raises the Slow Mode upper bound from 50.1 to 52.1 F1-score; hybrid inference preserves most speed while improving difficult cases.

5. Conclusion

LocateAnything reformulates visual grounding and detection around atomic geometric units with massive-scale training queries and on-demand inference. Its main limitation is reliance on supervised fine-tuning, leaving reinforcement learning for future optimization of decoding and difficult cases.

  • 5. Conclusion: LocateAnything unifies visual grounding and detection through Parallel Box Decoding, treating geometric elements as atomic units instead of 1D streams.This aligns training supervision with the coupled nature of spatial coordinates.
  • 5. Conclusion: 138M text-image training queries and flexible on-demand inference support LocateAnything’s performance across diverse tasks.The passage states that the framework delivered SOTA accuracy across diverse tasks, but its throughput value is truncated.
  • 5. Conclusion: Reinforcement learning is proposed to optimize block-level decoding, reduce fallback frequency, and improve exploration in hard dense or long-tail cases.The authors identify this as future work that could improve robustness and worst-case decoding speed.

A. Training and Inference Configurations … B.1. Leveraging Existing Open-Source Data

The paper details a four-stage training pipeline, efficient infrastructure for dual-formulation training, and an inference procedure with parallel decoding and cache management. It also constructs LocateAnything-Data by normalizing diverse open-source datasets, expanding grounding annotations, and adding negative queries to reduce hallucinations.

  • A.1.3. Stream Packing: Stream Packing dynamically combines variable-length samples into densely packed sequences, such as a 36,864-token target budget, to reduce padding waste and improve GPU utilization.This addresses sequence-length variability caused by block-wise expansion in the NTP+MTP design.
  • A.1. Training Details: LocateAnything uses a four-stage pipeline: two stages pretrain the base VLM for world knowledge, followed by two-stage SFT for detection and grounding.The initial alignment excludes detection and bounding-box grounding data; later fine-tuning incorporates 138M queries.
  • A.1.1. Base VLM Training (World Knowledge Injection): Base VLM pretraining begins with caption-related data in Stage 1 and integrates the full diverse dataset mixture in Stage 2 for broad multimodal capabilities.This alignment phase is explicitly designed to establish world knowledge before specialized detection and grounding training.
  • A.1.2. LocateAnything Fine-Tuning (Detection and Grounding Enhancement): LocateAnything fine-tuning uses comprehensive detection and grounding data, then reduces general data to 20% while increasing dense-scene data and maintaining a 25,600-token sequence length.The learning rate is decayed to 1 × 10−5, and all components remain trainable.
  • A.1.4. MagiAttention for Heterogeneous Mask Training: MagiAttention efficiently supports the irregular heterogeneous masks produced by combining stream packing with NTP causal attention and MTP block-causal or bidirectional patterns.Together, the two techniques improve token-level GPU utilization while handling distributed ultra-long-context attention correctly.
  • A.2. Inference Details: Inference uses nucleus sampling, KV caching, semi-autoregressive generation, and box-aware decoding across the reported evaluation configurations.The pipeline description covers generation modes, decoding strategies, and associated hyperparameters.
  • A.2.1. Generation Hyperparameters: A block size of 6 lets each MTP step predict up to 6 tokens simultaneously, with temperature 0.7, top-p 0.9, repetition penalty 1.1, and up to 8,192 new tokens.Inference also enables KV cache throughout and evaluates models in BF16 precision.
  • A.2.2. KV Cache Management: After each MTP step, KV-cache truncation removes mask tokens and the duplicated anchor token, retaining only committed positions to preserve training–inference causal consistency.Subsequent steps therefore attend only to the ground-truth generation history.

B.2. Multi-Targets Grounding Data Engine · B.3. Task-Specific Prompt Design

The framework builds a large-scale multi-target grounding dataset by synthesizing annotations from labeled detection datasets and unlabeled images, with verification for annotation quality. Task-specific prompts unify diverse grounding and detection tasks by mapping natural-language instructions to bounding-box or point outputs.

  • B.2. Multi-Targets Grounding Data Engine: The data engine automatically synthesizes multi-target grounding annotations from labeled detection data and unlabeled images to address limited dataset scale and diversity.This construction is illustrated in Fig. 9.
  • B.2. Multi-Targets Grounding Data Engine: For each ground-truth detection box, its category prompts Qwen3-VL to generate object-centric queries containing attributes, spatial relations, and reasoning cues.The source detection datasets include Open Images and Objects365.
  • B.2. Multi-Targets Grounding Data Engine: Molmo predicts candidate points for these queries, and points inside the corresponding ground-truth boxes are retained as reliable grounding supervision.The known ground-truth boxes provide the criterion for filtering candidate points.
  • B.2. Multi-Targets Grounding Data Engine: For unlabeled Unsplash and SA-1B images, Qwen3-VL generates diverse queries, Molmo predicts points, and SAM 3 converts those points into bounding boxes.This pathway expands the diversity of grounding targets.
  • B.2. Multi-Targets Grounding Data Engine: Qwen3-VL post-checks all generated boxes and filters inconsistent predictions to improve annotation quality.The verification stage is applied after box generation.
  • B.3. Task-Specific Prompt Design: Task-specific textual prompts support region-based tasks such as Object Detection, Text Grounding, Scene Text Detection, and Document… For complexThe framework specifies corresponding output formats and question templates for its versatile perception tasks.
  • B.3. Task-Specific Prompt Design: Phrase Grounding and GUI Grounding can predict single or multiple boxes or points depending on user intent.Prompt templates use [PHRASE] for free-form descriptions and [CATEGORIES] for comma-separated target category names.
  • B.3. Task-Specific Prompt Design: The unified prompting strategy connects natural-language instructions with precise spatial coordinate decoding across the supported perception tasks.The prompt vocabulary distinguishes free-form phrases from category lists.

B.4. Data Statistics and Distribution · C. Additional Experiments

The dataset spans six domains, exceeding 139M queries and 22M negative samples while exhibiting substantial variation in multi-target structure and query language. Additional experiments evaluate object pointing across diverse benchmarks using F1-scores in Hybrid Mode.

  • B.4. Data Statistics and Distribution: Over 139M queries and more than 22M negative samples comprise the collected dataset across six domains.Table 10 summarizes query, negative-sample, target, category, and query-length statistics.
  • B.4. Data Statistics and Distribution: Target counts per query vary substantially across domains and follow a long-tailed distribution.Most queries involve few targets, while a non-negligible portion contains many instances.
  • B.4. Data Statistics and Distribution: Query lengths vary across domains, reflecting different grounding paradigms and language patterns for describing visual targets.The reported length excludes template text and measures the words carrying actual target-description content.
  • B.4. Data Statistics and Distribution: The combined statistics characterize the dataset as large-scale, diverse, and multi-target, supporting training for heterogeneous visual domains and complex language queries.These properties are presented as the foundation for handling varied visual and linguistic grounding requirements.
  • C. Additional Experiments: Object pointing is evaluated across COCO, LVIS, Dense200, VisDrone, RefCOCOg, and HumanRef using F1-scores as the primary metric.The reported results use Hybrid Mode.
  • C. Additional Experiments: The additional experiment covers both conventional detection datasets and referring-grounding benchmarks in a single object-pointing evaluation.The benchmark set includes COCO, LVIS, Dense200, VisDrone, RefCOCOg, and HumanRef.

C.1. Results on Pointing Tasks · C.2. Comprehensive Performance Across Decoding Modes

LocateAnything-3B achieves state-of-the-art point-based localization across diverse benchmarks, including strong performance in dense scenes and complex human-intent grounding. Its Fast, Hybrid, and Slow decoding modes provide configurable trade-offs between geometric precision and inference latency.

  • C.1. Results on Pointing Tasks: LocateAnything-3B achieves state-of-the-art results across a diverse suite of point-based localization benchmarks under Hybrid Mode.The tasks require predicting a point inside the target’s bounding box or segmentation mask.
  • C.1. Results on Pointing Tasks: 84.7 F1@Point on HumanRef shows strong alignment of complex human intents to spatial regions.This result is reported alongside performance on COCO, Dense200, and RefCOCOg.
  • C.2. Comprehensive Performance Across Decoding Modes: LocateAnything provides three on-demand decoding modes—Fast, Hybrid, and Slow—for dynamically balancing geometric precision and inference latency.Their comparative performance is summarized in Tab. 12.

C.3. Backbone Generalization

On Qwen3-VL-4B trained exclusively on COCO, Parallel Box Decoding improves the speed–accuracy trade-off, indicating that its benefits are not tied to one backbone architecture.

  • Backbone Generalization: The study instantiates PBD on Qwen3-VL-4B and trains exclusively on COCO to isolate decoding effects from large-scale data scaling.This follows the controlled ablation setting used in the main paper.
  • Backbone Generalization: 50.8 to 52.0 COCO F1 and 2.8 to 9.4 BPS: the Hybrid configuration improves both localization quality and throughput.The comparison is reported for Qwen3-VL-4B in Table 13.
  • Backbone Generalization: PBD consistently improves the speed–accuracy trade-off on Qwen3-VL-4B, showing that box-aligned parallel decoding benefits are not backbone-specific.The conclusion is based on the controlled COCO experiment.

C.4. Mode Analysis and Throughput · C.5. Experimental Setup · C.6. Qualitative Comparisons

LocateAnything’s decoding modes trade off localization precision and latency, with Hybrid Mode providing production-oriented adaptive decoding at 12.7 BPS. Under standardized evaluation settings, qualitative comparisons show strong compositional, dense-object, and text localization with coherent bounding boxes.

  • C.4. Mode Analysis and Throughput: Slow Mode uses standard autoregressive generation and consistently establishes the upper bound for localization accuracy.The passage presents Slow Mode as next-token prediction.
  • C.4. Mode Analysis and Throughput: Hybrid Mode defaults to parallel decoding and selectively falls back to autoregressive generation when spatial ambiguity or format irregularities are detected.This adaptive strategy is described as the optimal choice for production pipelines.
  • C.4. Mode Analysis and Throughput: 12.7 BPS is achieved by Hybrid Mode while preserving the speed gains of parallelization and maintaining precise outputs.Throughput is reported in Boxes Per Second.
  • C.5. Experimental Setup: Throughput values are measured in Boxes Per Second on COCO, establishing a consistent baseline for speed comparison.The configuration is specified for throughput benchmarking.
  • C.5. Experimental Setup: COCO and LVIS images use an 840-pixel short side, while all other benchmarks use the model’s original image resolution.These input-resolution settings define the reported evaluation configurations.
  • C.6. Qualitative Comparisons: In REC, LocateAnything aligns nuanced free-form spatial and attribute-based human intents with the correct visual regions more effectively than Qwen3-VL and Rex-Omni.The qualitative comparison characterizes this as superior compositional grounding.
  • C.6. Qualitative Comparisons: In dense overlapping scenes, LocateAnything maintains compact, well-separated, accurate boxes, avoiding omissions and boundary ambiguity seen in Qwen3-VL and Rex-Omni.Examples include stacked logs and abacus beads.
  • C.6. Qualitative Comparisons: For OCR, LocateAnything tightly bounds text elements, while Hybrid Mode fallback handles complex layouts without sacrificing structural coherence.Baseline models frequently show format irregularities or merge distinct text blocks.
Loading 2605.27365v2…