PaperlayerView exact revision on arXiv

Source-linked AI summary

Detect Anything via Next Point Prediction

Qing Jiang, Junan Huo, Xingyu Chen, Yuda Xiong, Zhaoyang Zeng, Yihao Chen, Tianhe Ren, Junzhi Yu, Lei Zhang

arXiv:2510.12798v1cs.CV

TL;DR

MLLM-based object detection remains limited by language understanding, coordinate discretization, and teacher-forced behavior. Rex-Omni addresses these issues with unified quantized coordinate prediction, custom data engines, and SFT plus GRPO training, achieving competitive or superior zero-shot perception across diverse tasks. Its results support a language-aware visual perception framework, while SFT-related coordinate and output-regulation deficiencies remain important limitations addressed by post-training.

  • Problem

    MLLM-based detectors struggle with complex semantic descriptions, discrete coordinate learning, and autonomous output regulation compared with regression-based detectors.

  • Method

    Rex-Omni unifies visual perception as coordinate-sequence generation using quantized coordinate tokens, custom data engines, and a two-stage SFT plus GRPO training pipeline.

  • Results

    Rex-Omni achieves state-of-the-art or highly competitive zero-shot performance across a wide array of visual perception tasks.

  • Takeaways & Limitations

    Rex-Omni demonstrates a unified framework combining precise localization with robust language understanding across diverse perception tasks.

  • Takeaways & Limitations

    SFT can misalign coordinate-token loss with continuous geometry and cause missed or repetitive detections by fixing training outputs to ground-truth counts.

Abstract

from arXiv · show

Object detection has long been dominated by traditional coordinate regression-based models, such as YOLO, DETR, and Grounding DINO. Although recent efforts have attempted to leverage MLLMs to tackle this task, they face challenges like low recall rate, duplicate predictions, coordinate misalignment, etc. In this work, we bridge this gap and propose Rex-Omni, a 3B-scale MLLM that achieves state-of-the-art object perception performance. On benchmarks like COCO and LVIS, Rex-Omni attains performance comparable to or exceeding regression-based models (e.g., DINO, Grounding DINO) in a zero-shot setting. This is enabled by three key designs: 1) Task Formulation: we use special tokens to represent quantized coordinates from 0 to 999, reducing the model's learning difficulty and improving token efficiency for coordinate prediction; 2) Data Engines: we construct multiple data engines to generate high-quality grounding, referring, and pointing data, providing semantically rich supervision for training; \3) Training Pipelines: we employ a two-stage training process, combining supervised fine-tuning on 22 million data with GRPO-based reinforcement post-training. This RL post-training leverages geometry-aware rewards to effectively bridge the discrete-to-continuous coordinate prediction gap, improve box accuracy, and mitigate undesirable behaviors like duplicate predictions that stem from the teacher-guided nature of the initial SFT stage. Beyond conventional detection, Rex-Omni's inherent language understanding enables versatile capabilities such as object referring, pointing, visual prompting, GUI grounding, spatial referring, OCR and key-pointing, all systematically evaluated on dedicated benchmarks. We believe that Rex-Omni paves the way for more versatile and language-aware visual perception systems.

1. Introduction

Rex-Omni addresses the limited language understanding and coordinate-prediction difficulties of MLLM-based detection with unified coordinate prediction, specialized data engines, and two-stage training. It achieves strong zero-shot perception across detection and broader visual tasks.

  • Motivation: Open-vocabulary detectors can identify arbitrary categories but remain limited in handling complex semantic descriptions.Grounding DINO may detect all apples when prompted for red apples, reflecting shallow language understanding.
  • Motivation: Existing MLLM detectors rarely match regression-based detectors because discrete coordinate prediction and teacher-forced training create learning and inference mismatches.Cross-entropy treats nearby coordinate tokens as equally wrong, while teacher forcing exposes the model only to ground-truth prefixes.
  • Contribution: Rex-Omni is a 3B-scale MLLM designed to overcome these limitations while combining language understanding with visual perception.The paper presents this model as the proposed response to the identified formulation and training challenges.
  • Core design: Rex-Omni unifies pointing, detection, polygons, and keypoints as coordinate-sequence generation using 1,000 quantized coordinate tokens.The formulation maps coordinate values to tokens from 0 to 999, reducing coordinate-learning complexity and improving spatial token efficiency.
  • Results: Rex-Omni achieves superior zero-shot F1-score performance on COCO compared with traditional detectors and other MLLMs.The cited comparison includes DINO-ResNet50, Grounding DINO, and SEED1.5-VL.
  • Results: Across detection, referring, prompting, GUI and layout grounding, OCR, pointing, keypointing, and spatial referring, Rex-Omni consistently outperforms traditional detectors and MLLMs.These tasks are handled through direct coordinate-point prediction.
  • Core design: Rex-Omni combines direct prediction with quantized relative coordinates represented by special tokens for efficient spatial encoding.The design philosophy contrasts direct coordinate prediction with retrieval-based and external-decoder alternatives.

2. Task Formulation

Rex-Omni formulates visual perception as a unified text-based interface that predicts quantized coordinate tokens and structured outputs across diverse tasks. Its relative-coordinate design uses dedicated vocabulary tokens to reduce coordinate-learning complexity and improve token efficiency.

  • 2.1. Coordinate Formulation: Relative coordinates constrain prediction to a bounded range of 1,000 categories, reducing the learning complexity of coordinate classification.The formulation is selected over absolute coordinates because it limits the coordinate prediction range.
  • 2.1. Coordinate Formulation: A bounding box requires four special tokens rather than 15 atomic tokens, improving token efficiency and inference speed in dense scenes.The comparison includes separators in the 15-token representation.
  • 2.2. Input Format: All visual perception tasks use a unified natural-language query interface specifying target objects or relationships in the image.This interface supports instruction-driven integration of diverse vision-language tasks.
  • 2.2. Input Format: Visual prompting converts an indicated box into quantized coordinate tokens, then instructs the model to find objects sharing its category.This handles objects that are difficult to describe linguistically while retaining Rex-Omni’s text-based interface.
  • 2.3. Output Format for Each Task: Outputs pair descriptive phrases with coordinate sequences inside demarcation tokens, with missing targets represented by None.The format supports boxes, points, polygons, and structured JSON keypoints.
  • 2.3. Output Format for Each Task: Boxes use [x0, y0, x1, y1] coordinates sorted by x0, points use [x0, y0] pairs, polygons use coordinate sequences, and keypointing returns boxes with keypoints.These task-specific coordinate structures are embedded within the common phrase-and-demarcation format.
  • 2.1. Coordinate Formulation: Rex-Omni uses relative coordinates quantized to 1,000 values, represented by dedicated special tokens in the language-model vocabulary.The model repurposes the final 1,000 vocabulary tokens without introducing additional parameters.

4. Training Pipelines

Rex-Omni uses supervised fine-tuning to establish coordinate prediction, followed by GRPO reinforcement post-training that improves geometric precision and regulates output behavior. Geometry-aware rewards and task-specific reward functions address SFT’s discretization and behavioral limitations.

  • Stage 1: Supervised Fine-Tuning: 22 million annotated samples are used for teacher-guided SFT to establish fundamental coordinate prediction capabilities.The model first learns to map quantized coordinate tokens to image positions through large-scale supervised training.
  • Limitations of SFT: SFT’s cross-entropy coordinate training misaligns token classification with continuous geometry, treating small and severe coordinate errors inconsistently.A near-correct token can receive the same penalty as a substantially misaligned coordinate, while a severely wrong box may incur only one token error.
  • Limitations of SFT: Teacher-forced SFT fixes output length to the ground-truth count, causing missed detections or repetitive predictions during autonomous inference.Because the model is not trained to regulate output quantity, it may under-generate or produce duplicate or slightly shifted boxes.
  • Stage 2: GRPO-Based Post-Training: GRPO post-training samples complete responses, normalizes rewards within each group, and uses clipped policy optimization with KL regularization.The KL penalty keeps the policy from diverging excessively from the SFT reference model.
  • Stage 2: GRPO-Based Post-Training: Variable-length reward-guided optimization encourages accurate, non-redundant predictions and mitigates SFT-induced repetition or over-generation.Repetitive outputs receive lower rewards, allowing the model to learn output regulation beyond teacher-forced behavior.
  • Geometry-Aware Rewards: Geometry-aware rewards use IoU, point-in-mask, and point-in-box criteria to assess spatial correctness across box- and point-based perception tasks.The reward design covers detection, grounding, referring, OCR, pointing, and GUI grounding, while the F1-style signal balances over- and under-prediction.

5. Benchmark Results

Rex-Omni is evaluated on common, long-tailed, and dense object detection benchmarks using zero-shot and controlled prompting settings. It performs strongly on common and long-tailed detection, while dense tiny-object scenes expose remaining failure modes that GRPO post-training substantially reduces.

  • Common object detection: COCO evaluation uses Recall, Precision, and F1 rather than AP, with Rex-Omni compared against MLLMs and regression-based detectors.The evaluation includes Rex-Omni-SFT and the full Rex-Omni model, with deterministic sampling for Rex-Omni variants.
  • Common object detection: Among MLLMs, Rex-Omni surpasses existing approaches, including the previously strongest SEED1.5-VL.
  • Common object detection: At IoU=0.5, Rex-Omni outperforms Grounding DINO-SwinT and DINO-R50 in zero-shot evaluation.At IoU=0.95, it only marginally outperforms DAB-DETR, indicating a narrower advantage for extremely tight boxes.
  • Long-tailed object detection: On LVIS, MLLMs generally outperform open-set detectors, while Rex-Omni achieves competitive F1 at IoU=0.5 and state-of-the-art mIoU.The full model also improves substantially over Rex-Omni-SFT.
  • Dense and tiny object detection: Dense and tiny-object detection remains difficult for MLLMs, producing large-box and structured-duplicate prediction failures.After GRPO post-training, duplication issues largely disappear.

5.4. Referring Object Detection

The section evaluates language- and example-conditioned localization, including referring detection and visual prompting. Rex-Omni is strong across these settings, although it remains below T-Rex2 on visual prompting.

  • Referring object detection: Referring detection requires fine-grained language understanding and alignment between natural-language expressions and visual objects.
  • Referring object detection: HumanRef is more challenging than RefCOCOg because expressions can map to multiple boxes and require attribute, position, interaction, or reasoning cues.HumanRef averages two ground-truth boxes per expression, whereas RefCOCOg maps each expression to one box.
  • Referring object detection: MLLMs consistently excel at referring detection, while Grounding DINO underperforms across the evaluated benchmarks.Rex-Omni ranks second only to SEED1.5-VL on HumanRef.
  • Visual prompting: Visual prompting detects objects matching categories indicated by example bounding boxes when language descriptions are insufficient.The evaluation uses FSC147, COCO, LVIS, and Dense200, with F1 for detection and MAE for counting.
  • Visual prompting: Rex-Omni shows strong visual prompting in dense and long-tailed scenes but remains below the specialist T-Rex2 overall.

5.6. Object Pointing

Rex-Omni’s point-based formulation is evaluated across object pointing, GUI grounding, and layout grounding. It achieves strong or leading results while retaining an advantage in open-set language-conditioned localization.

  • Object pointing: Object pointing predicts a precise point within a specified target object rather than a bounding box.The evaluation spans common, long-tailed, dense, tiny-object, and referring-expression datasets.
  • Object pointing: Rex-Omni attains the highest F1-scores across general and challenging pointing datasets, despite MLLMs struggling with dense small instances.
  • GUI grounding: GUI grounding evaluates point predictions for UI elements across mobile, desktop, web, and ultra-high-resolution interfaces.A prediction is correct when the point falls inside the target bounding box.
  • GUI grounding: Among 3B-parameter models, Rex-Omni achieves the highest accuracy on both ScreenSpot-V2 and ScreenSpot-Pro.
  • Layout grounding: Layout grounding combines localization of document elements with interpretation of their spatial and semantic relationships.The evaluation covers DocLayNet and the more diverse M6Doc dataset.
  • Layout grounding: Rex-Omni outperforms other MLLMs by a large margin on layout grounding but remains behind closed-set models.Its open-set formulation supports generalization to unseen domains and novel layout structures.

5.9. OCR

Rex-Omni is evaluated on OCR, spatial referring, and keypoint detection using its coordinate-based outputs. Results show strong text localization, spatial reasoning, and cross-domain keypoint generalization, with task-specific trade-offs against specialists.

  • OCR: OCR is evaluated as detection plus exact text recognition, using F1 across bounding-box and polygonal outputs.A prediction must match the text region and exactly match the recognized text.
  • OCR: Rex-Omni’s bounding-box OCR significantly outperforms SEED1.5-VL across datasets and is comparable or superior to PaddleOCRv5 in several aspects.Its polygonal output is also competitive, with leading results on ICDAR2015 after GRPO post-training.
  • Spatial pointing: Spatial grounding requires interpreting relational cues such as relative position, anchoring, and free-space placement.RefSpatial-Bench evaluates location and placement using precise mask annotations.
  • Spatial pointing: Rex-Omni substantially outperforms prior proprietary and referring-specialist models on both RefSpatial tasks.The results also indicate applicability to scenarios requiring accurate grasping and placement.
  • Keypoint detection: COCO and AP10K evaluate human and animal keypoint detection using F1 at OKS thresholds of 0.5, 0.95, and their mean.
  • Keypoint detection: Rex-Omni has more balanced human- and animal-keypoint results than X-Pose, outperforming it substantially on AP10K despite lower COCO scores.The full model also consistently improves over Rex-Omni-SFT.

6. In-depth Analysis of Rex-Omni

The analysis shows that GRPO improves Rex-Omni beyond SFT by rapidly unlocking latent capabilities and correcting duplicate and large-box behaviors. Its benefits are especially pronounced on complex tasks, while coordinate tokenization improves output efficiency.

  • Training Dynamics: GRPO-enhanced Rex-Omni consistently outperforms the SFT-only model across coordinate prediction benchmarks.The analysis compares performance across SFT and GRPO training stages and benchmarks.
  • Training Dynamics: GRPO produces a rapid performance jump with few training steps, unlike SFT’s gradual improvement and post-training plateau.The improvement is attributed to behavior-aware rewards and sequence-level feedback rather than simply greater data exposure.
  • Behavioral Correction via GRPO: GRPO suppresses duplicate predictions by assigning low rewards to repeated coordinate sequences generated autonomously.SFT-trained models show larger gains after duplicate removal, whereas GRPO models show minimal gains.
  • Behavioral Correction via GRPO: GRPO reduces large-box failures on Dense200, with large boxes falling from 20.5% of SFT predictions to 3.5%.After removal, F1@IoU=mIoU rises from 44.9 to 56.7 for SFT and from 58.3 to 60.0 for GRPO.
  • Prediction Quality: GRPO’s role varies with task complexity: it improves sampling consistency on simpler datasets and prediction quality on more challenging ones.On COCO, SFT-Sampling-Vote reaches 72.6 F1@0.5 versus 72.0 for GRPO and 68.2 for base SFT; on LVIS and Dense200, sampling does not approach GRPO.
  • Coordinate Representation Efficiency: Rex-Omni requires 7.6 tokens per box on COCO on average, compared with 148 for SEED1.5-VL.The comparison measures boxes per image, output tokens per image, and tokens per box on sampled COCO and Dense200 images.
  • Limitations: Inference speed remains limited because runtime scales directly with the number of detected objects.Detecting hundreds of objects can exceed 16 seconds, whereas detecting a few objects takes under 2 seconds.

7. Conclusion

Rex-Omni is a 3B-parameter MLLM designed to address MLLM-based object-detection challenges through efficient coordinate tokenization, custom data generation, and SFT+GRPO training. It achieves state-of-the-art or highly competitive zero-shot performance across diverse visual perception tasks, while inference speed remains a limitation.

  • Rex-Omni combines special-token coordinate tokenization, custom data engines, and a two-stage SFT+GRPO training pipeline.
  • Rex-Omni achieves state-of-the-art or highly competitive zero-shot performance across a wide array of visual perception tasks.
  • GRPO-based post-training corrects SFT-induced duplicate and large-box prediction behaviors.
  • Inference speed remains a limitation, motivating future work on model acceleration and reward-guided sampling.

8. Related Work

Related work spans regression-based, open-set, and MLLM-based object detectors. MLLMs offer stronger language understanding, but existing methods often struggle with fine-grained spatial precision and repetitive or misaligned predictions.

  • Regression-based Object Detection Methods: Regression-based detectors predict bounding-box properties such as center coordinates and dimensions as normalized offsets.
  • Open-set Object Detection Methods: Open-set detectors target arbitrary object categories without task-specific fine-tuning, often using CLIP or BERT to align text and visual representations.
  • Open-set Object Detection Methods: Visual prompts help recognize rare or hard-to-describe objects but are less general than text prompts.
  • MLLM-based Object Detection Methods: MLLM-based detectors reframe detection as next-token prediction over discrete, quantized coordinate sequences.
  • MLLM-based Object Detection Methods: Existing MLLM-based methods often suffer from low recall, coordinate drift, spurious duplicates, and inadequate fine-grained spatial precision.

A.1. More Visualization Results

Additional visualizations show Rex-Omni across common and long-tailed detection, dense detection, referring, pointing, layout grounding, and OCR tasks. These examples complement the paper’s quantitative evaluations.

  • The visualization appendix covers common and long-tailed object detection, dense object detection, referring, pointing, layout grounding, and OCR.
  • Figure 19 presents visualizations for common and long-tailed object detection.
  • Figure 20 presents visualizations for dense object detection.
  • Figures 21–23 present visualizations for object referring, object pointing, and layout grounding.
  • Figure 24 presents visualizations for OCR.
Loading 2510.12798v1…