PaperlayerView exact revision on arXiv

Source-linked AI summary

GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitives

Qize Yu, Lianrui Fan, Boyu Chen, Jiaqi Liang, Xini Ding, Yue Chen, Zetian Song, Yuran Wang, Yi Zou, Kaixuan Wang, Tianxing Chen, Wenxuan Song, Bohan Zhou, Mingleyang Li, Siqiao Huang, Yuqi Ye, Caigao Jiang, Wei Wei, Ruihai Wu, Hang Zhang, Yixiao Ge, Shuchang Zhou, Shilong Liu, Xianming Liu, Ping Luo, Shiyu Huang

arXiv:2609.39601v1cs.CVcs.AIcs.LGcs.RO

TL;DR

Precise, task-conditioned grounding remains a challenge for the general-purpose backbones used in physical-intelligence models. GroundingPI represents points and boxes as quantized coordinates in a shared vocabulary and is trained for broad perception. It sets a grounding benchmark state of the art and shows downstream manipulation gains.

  • Problem

    General-purpose backbones used by physical-intelligence models have weaknesses in demanding precise-grounding settings, including dense scenes and tiny objects.

    Source: §1 · p. 4

  • Method

    GroundingPI is a 4B grounding model that generates points and boxes as quantized coordinates in a shared vocabulary for language-conditioned perception.

    Source: §1 · p. 4, §3.1 · p. 6

  • Results

    GroundingPI averages 73.68% across 34 grounding benchmarks, above GPT-6 Astra (71.54%), and improves manipulation transfer, including all four evaluated OOD settings.

    Source: §1 · p. 4

  • Takeaways & Limitations

    The findings support dedicated grounding as a perceptual foundation for manipulation generalization and action-data efficiency, with dense grounding valuable and OCR a potential catalyst.

    Source: §6 · p. 18

    Backbone comparisons do not establish that architecture causes the observed rankings.

    Source: §5.3.3 · p. 16

Abstract

from arXiv · show

Precise grounding matters. It specifies which object is the target and where that object is, even in clutter and for tiny objects, and it has to be fast enough for closed-loop control. Yet vision-language-action (VLA) and world-action models (WAMs) take perception from general-purpose vision-language and video-generation backbones, which still fail in these settings. We introduce GroundingPI, a 4B grounding foundation model that generates points and boxes as quantized coordinates in a shared vocabulary. Training combines multimodal and spatial pretraining, supervised fine-tuning, and reinforcement learning with GRPO, using supervision from public datasets and dedicated data engines. Against 44 baselines across 34 grounding benchmarks spanning 11 perceptual capabilities, GroundingPI establishes a new state of the art, averaging 73.68%, above the larger GPT-6 Astra (71.54%). As a downstream visual backbone, GroundingPI improves performance on robotic manipulation and autonomous driving. On RoboTwin 2.0, it outperforms every mainstream backbone we evaluate in all four out-of-distribution settings, by up to 24.8% relative to the strongest backbone. On RoboCasa-GR1, GroundingPI trained with 50% of the demonstrations outperforms those baselines trained with 75%. On nuScenes, used as the visual backbone, GroundingPI attains an average open-loop L2 error of 0.296 m. We systematically analyze GroundingPI's pretraining in scale and data composition. Downstream autonomous driving and robotic manipulation improve as the pretraining is scaled. Analyzing the data recipe across these 11 perceptual capabilities shows dense grounding's substantial benefits for both, and OCR's potential as a catalyst for perceptual learning. These results support grounding as a perceptual foundation, and dedicated perceptual pretraining as a promising direction for foundation models of physical intelligence.

1 Introduction

GroundingPI targets the precise, language-conditioned perception that general-purpose backbones may lack for physical interaction. The paper presents a shared-vocabulary grounding model, its data and training pipeline, and evaluations of grounding and downstream action learning.

1 Introduction

  • General-purpose VLM and video-generation backbones are not optimized for precise interaction grounding, with weaknesses in dense scenes and tiny-object settings.The paper also notes compute and latency constraints on relying on larger backbones for action execution.Source: §1 · p. 4

3.1 Model Architecture and Grounding Formulation

  • GroundingPI is a 4B autoregressive model that uses a visual encoder, projector, and language decoder to produce structured perception outputs with quantized coordinates in a shared vocabulary.Its visual primitives include points and bounding boxes, and queried categories absent from an image can yield None.Source: §1 · p. 4, §3.1 · p. 6

1 Introduction

  • Across 34 grounding benchmarks, GroundingPI averages 73.68%, above larger GPT-6 Astra (71.54%), and improves manipulation transfer, including all four OOD settings and RoboCasa-GR1 with 50% of demonstrations.The RoboCasa-GR1 comparison is against baselines trained with 75% of demonstrations.Source: §1 · p. 4
  • Pretraining scale and data composition matter for grounding and downstream transfer; dense grounding benefits driving and manipulation, while OCR may catalyze perceptual learning.The authors frame these insights as relevant to System-1-style execution and future embodied foundations.Source: §1 · pp. 4–5

2.1 Visual Grounding Models; 2.2 Foundation Models for Physical Intelligence; 2.3 Perceptual Learning for Action Transfer

  • Prior policy recipes add selected perceptual proxies, while VLA, WAM, and embodied models may lack precise task-conditioned perception; the paper motivates grounding learned natively before action adaptation.Related work also traces generative grounding to structured sequences of labels and quantized coordinates shared across perception tasks.Source: §2.3 · p. 5, §2.2 · p. 5, §2.1 · p. 5

3.2 GroundingPI Data

  • Training data combines public datasets with a data engine that fuses field-level teacher annotations, validates labels by task, and iteratively resolves uncertain cases.Accepted labels train a grounding expert, while complementary teachers and local observations help expand and refine supervision.Source: §3.2 · p. 6

3.3.1 Base VLM Training; 3.3.2 Supervised Fine-Tuning; 3.3.3 Reinforcement Post-Training

  • Training progresses through multimodal and visual/video pretraining, supervised fine-tuning of structured grounding outputs, and GRPO post-training with grounding- and OCR-specific rewards.SFT uses assistant-response labels including protocol markers and coordinates; GRPO rewards address coverage, localization, text–geometry consistency, and output reliability.Source: §3.3.1 · pp. 6–7, §3.3.2 · p. 7, §3.3.3 · p. 7

4 GroundingPI Model towards Physical Intelligence; 4.1 Autonomous Driving; 4.2 Robot Manipulation

  • For downstream transfer, GroundingPI serves as a shared perceptual foundation for trajectory prediction and robot manipulation, with controlled comparisons varying the pretrained backbone while holding action-learning setup fixed.Driving predicts six ordered waypoints over three seconds; manipulation uses a π-style flow-matching policy conditioned on visual-language features and robot state.Source: §4 · p. 7, §4.1 · p. 8, §4.2 · p. 8

5 Experiments

This section evaluates GroundingPI’s grounding performance and transfer to driving and manipulation, alongside how pretraining scale, data composition, base-model choices, and design choices relate to downstream performance. It also frames perception-native models as a foundation for reliable execution, with some architectural explanations remaining hypotheses.

5.1 Grounding Benchmark Results

  • 73.68% Avg makes GroundingPI the top reported grounding model, above GPT-6 Astra at 71.54%.Across selected tasks, GroundingPI reaches 74.53 on Dense200 versus Astra’s 65.04 and Qwen3-VL-4B’s 14.02; it also reaches 72.47 on SROIE and 74.82 on M6Doc, while Astra remains stronger on TotalText and SenseNova-Vision slightly leads on DocLayNet.Source: §5.1 · p. 9, Figure 5 · p. 9, Table 1 · p. 10
  • GroundingPI reaches 72.47 on SROIE and 74.82 on M6Doc, but Astra leads Dense200 pointing and remains stronger on TotalText, while SenseNova-Vision slightly leads on DocLayNet.GroundingPI leads six of seven object-pointing columns; the results distinguish point selection from box-boundary precision.Source: §5.1 · p. 9, Table 3 · p. 11, Table 4 · p. 11

5.2 Physical Intelligence Performance

  • GroundingPI lowers nuScenes average open-loop L2 error to 0.296 m and has the highest manipulation success in five of six settings, including all four OOD settings.The nuScenes error compares with Qwen3-VL-4B at 0.301 m and RynnBrain at 0.308 m; the manipulation comparisons hold the action expert, data, and training budget fixed within each benchmark.Source: §5.2 · p. 11, §5.2.1 · p. 12, Figure 6 · p. 12

5.2.2 Robot Manipulation Performance

  • On RoboTwin2.0, GroundingPI ranks first on Full and every OOD setting, reaching 66.20% on Full and 17.60% on Clean2Random versus Rex-Omni’s 14.10%.It ranks first in all four OOD settings overall; on RoboCasa-GR1 Full it scores 37.75%, compared with RynnBrain’s 39.00%.Source: §5.2.2 · p. 13

5.2.3 Data Efficiency

  • At 50% of RoboCasa-GR1 demonstrations, GroundingPI achieves 28.75% SR, exceeding every baseline at 75%, where RynnBrain leads with 27.75%.The comparison varies action-data quantity while keeping the architecture, action-learning approach, and other training choices unchanged.Source: §5.2.3 · p. 14, Figure 8 · p. 14

5.3.1 Effects of Grounding Training

  • As grounding pretraining grows from 88.4B to 221B tokens, grounding scores and RoboCasa-GR1 manipulation performance improve on both ID and OOD splits.Across the scaling trajectory, action-learning gains track grounding gains, with generalization tracking grounding quality most closely.Source: §5.3.1 · p. 15, Figure 9 · p. 15

5.3.2 What Transfers from Grounding to Action?

  • Removing dense grounding causes larger losses in manipulation and nuScenes performance than removing referring or robo pointing, while the full supervision mixture performs best across manipulation and driving.Removing the OCR-containing group causes the largest manipulation losses on both splits, 5.33/3.63 pp, although OCR supervision alone transfers poorly; its proposed role as a perceptual-learning catalyst remains a hypothesis.Source: §5.3.2 · pp. 15–16, Figure 10 · p. 16

5.3.3 Discussion

  • RynnBrain1.1 trails RynnBrain in all six manipulation settings despite improving driving, but the comparisons do not establish architectural causes of the rankings.The discussion hypothesizes that explicit localization may encourage fine-grained spatial evidence, with relative depth offering complementary supervision without requiring absolute scale.Source: §5.3.3 · pp. 16–17
  • The discussion proposes broad reasoning foundations for System 2 planning and perception-native foundations for reliable System 1 execution, while noting that a deployed dual-system controller and its latency remain unevaluated.The proposed advantage over absolute metric supervision also remains untested.Source: §5.3.3 · p. 17

5.3.4 Ablation Study

  • GroundingPI uses 7.6/5.1 tokens per box on COCO/Dense200, versus 148.8/74.5 for SEED1.5-VL; this cross-model comparison measures serialization, not matched latency.Source: §5.3.4 · p. 17, Table 5 · p. 18

6 Conclusion

  • Across 34 grounding benchmarks, GroundingPI outperforms similarly sized models, remains competitive with GPT-6 Astra, and transfers effectively to driving and manipulation.The supplied conclusion passages do not provide the benchmark score or downstream transfer metrics.Source: §6 · pp. 17–18

AI Use Statement; Research Scope, Data Use, and Institutional Disclaimer

  • The authors report using generative AI for code assistance, writing polish, figures, and pseudo-label generation, but not for research ideas, methodology, or experiment interpretation.They describe the work as research-only and state that findings do not imply commercial readiness, safety, or suitability for deployment.Source: AI Use Statement · p. 18, Research Scope, Data Use, and… · p. 19

Ethics Statement

  • The experiments use offline nuScenes evaluation and simulated robot manipulation; no real-world vehicles or robots were deployed.The authors recommend safety evaluation and human oversight before real-world embodied use.Source: Ethics Statement · p. 18

Reproducibility Statement

  • The authors will publicly release GroundingPI’s code, weights, evaluation data, training configurations, and evaluation scripts to support reproducible evaluation.The statement also points to documentation of protocols, data construction, training, downstream evaluation, and complete grounding results.Source: Reproducibility Statement · p. 19

A.1 Input/Output Protocol

  • GroundingPI uses one language-conditioned interface for box and point prediction, pairing semantic labels or transcriptions with geometric payloads.Boxes encode four quantized coordinates in xyxy order and points encode two in (x, y) order; coordinates are normalized to the image dimensions.Source: §A.1 · pp. 25–26

A.2 Architecture, Tokenizer, and Visual Processing

GroundingPI uses a shared language-model vocabulary to represent structured visual outputs, with a multimodal architecture that combines visual features and language generation.

A.2 Architecture, Tokenizer, and Visual Processing

  • GroundingPI combines a MoonViT-V2 (Kimi K3) visual encoder, a learnable multimodal projector, and a Qwen3-4B language backbone with full attention across its 36 language layers.Visual embeddings enter the language sequence, which uses one-dimensional rotary position indices.Source: §A.2 · p. 26
  • The architecture and visual-processing details are summarized in Table 8, whose positional capacity is distinct from the training sequence limit.Table 7 specifies canonical prompts and output geometry, with italic fields replaced by query-specific text or coordinates.Source: Table 8 · p. 26, Table 7 · p. 26
  • The tokenizer adds 1,000 coordinate tokens and a category separator to the 151,669-entry base vocabulary, for 152,670 entries total.Semantic, structural, and coordinate tokens share an output vocabulary head; input embeddings and output head are untied.Source: §A.2 · p. 27
  • The complete multimodal model contains approximately 4.844B parameters, including the visual encoder and vocabulary matrices.Qwen3-4B names the language-backbone family.Source: §A.2 · p. 27

A.3 Data Engine

The data engine fuses and validates annotations before deriving task-specific supervision, while training progresses from general multimodal learning to spatial specialization and reward-based post-training.

A.3 Data Engine

  • Accepted instance records are reused to derive grounding, referring, pointing, visual-prompting, and OCR supervision while retaining each task’s own required fields and coverage conditions.The data engine fuses semantic and geometric fields separately, validates query-dependent requirements, and uses targeted observations and expert revisions to resolve uncertainty.Source: §A.3 · p. 27

A.4 Base VLM Training (Pretrain 1)

  • Pretraining proceeds from projector alignment with image captions to joint text-image training and then broader visual, video, and instruction-following training.Stage 1 updates only the projector; Stage 2 unfreezes all modules, and Stage 3 increases context and packing lengths to 32,768 tokens.Source: §A.4 · pp. 27–28

A.5 Coordinate Alignment (Pretrain 2 / SFT)

  • Stage 4 specializes all modules on eight spatial task groups, training semantic labels, quantized point and box coordinates, and protocol markers with autoregressive supervision.Coordinate tokens use the same cross-entropy objective as other valid response tokens; no additional IoU regression loss is introduced.Source: §A.5 · p. 28

A.6 Reinforcement Learning (RL)

  • GRPO post-training starts from the coordinate-aligned SFT checkpoint and optimizes complete generated outputs using task-specific rewards.It samples eight responses per image-query pair; the vision encoder and projector remain frozen during reinforcement post-training.Source: §A.6 · pp. 28–29

A.6.1 Grounding Rewards

  • Grounding rewards combine set-coverage feedback with strict localization and output-quality terms, including detection F1 at IoU thresholds 0.50, 0.75, and 0.95.The set-completeness surrogate matches each reference independently and may reuse predictions; strict reward also scores localization, format, count, ordering, duplicates, and oversized boxes.Source: §A.6.1 · p. 30, Table 12 · p. 30

A.6.2 OCR Rewards

  • OCR scoring matches text instances one-to-one using box overlap and normalized transcription similarity, with hard and soft text-region agreement and count agreement contributing to view scores.Word and line views are scored as complete alternatives rather than combining isolated matches from incompatible views.Source: §A.6.2 · p. 30
  • OCR reward aggregation combines content, format, coverage, and penalties, with separate branches for ordinary and complex text arrangements.Curved, rotated, or complex text can use complementary reference views; all terms are combined into one scalar before within-prompt standardization.Source: §A.6.2 · p. 31

B Training Details and Evaluation Setup

The appendix specifies the driving and manipulation interfaces and the matched backbone-to-action protocol used to compare physical-intelligence foundations.

B.1 Autonomous Driving

  • Driving predictions use the current front-camera image and up to seven ego-state records to predict six cumulative future positions over three seconds.Positions use metric coordinates, and the reported open-loop metric averages errors up to each horizon rather than only endpoint errors; it does not establish closed-loop control performance.Source: §B.1 · pp. 31–32

B.2 Robot Manipulation

  • The manipulation comparison isolates the pretrained foundation by sharing a backbone-to-action interface and fixed-capacity action expert across VLA and WAM models.The compared foundations include vision-language, video-generation, grounding, and embodied models.Source: §B.2 · p. 32

B.2.1 Unified Backbone-to-Action Interface

  • Each foundation supplies eight hidden states sampled at normalized depths, projected to a common width and compressed into 64 condition tokens per depth.The same resampler is shared across depths and models, and each backbone produces all conditions in a single forward pass that is cached during action denoising.Source: §B.2.1 · p. 32

B.2.2 Fixed Layer-wise Action DiT

  • The common Action DiT has 16 alternating cross-attention and self-attention blocks, with eight blocks attending to layer-wise backbone conditions.Its action-side input includes encoded proprioception, planning tokens, and a noisy action chunk with its flow timestep; only action-token positions are decoded.Source: §B.2.2 · pp. 32–33, Table 14 · p. 33

B.2.3 Training Budget and Evaluation Protocol

The manipulation protocol matches downstream training conditions across foundations so the comparison focuses on how their representations support action learning.

B.2.3 Training Budget and Evaluation Protocol

  • All foundations use the same action demonstrations, sampling and augmentation, optimization settings, and task suites within each benchmark.RoboTwin 2.0 uses Clean and Randomized protocols for Full, and Clean training with Randomized evaluation for Clean2Random; RoboCasa-GR1 uses the same task suite and demonstration pool.Source: §B.2.3 · p. 33, Table 15 · p. 33
  • Training runs for 40,000 updates on 40 accelerators in bfloat16, with a global batch size of 1,280, AdamW (β1 = 0.95, β2 = 0.999, ϵ = 10−8), and cosine decay.The action expert and interface use a 10^-4 learning rate, while adapted backbone parameters use 10^-5; RoboCasa uses an identical demonstration pool across foundations at each data fraction.Source: §B.2.3 · p. 33
  • The protocol fixes action-head capacity, condition-token budget, state and action paths, data, optimizer, schedule, flow objective, and inference computation.Only the width-specific backbone projectors vary in size, and they remain small relative to the shared action expert.Source: §B.2.3 · pp. 33–34

C.1 Data Efficiency and the Burden on Action Demonstrations

With half the RoboCasa-GR1 demonstrations, GroundingPI exceeds the strongest baseline trained with three quarters; the scaling analysis tracks performance as grounding-training exposure increases while holding the action setup fixed.

C.1 Data Efficiency and the Burden on Action Demonstrations

  • 28.75% SR with half the RoboCasa-GR1 demonstrations exceeds RynnBrain’s 27.75% SR with three quarters, though RynnBrain leads at full data, 39.00% to 37.75%.The main advantage in this experiment is adaptation with limited demonstrations, not the highest full-data in-distribution score.Source: §C.1 · p. 34
  • The data-efficiency result concerns manipulation demonstrations; the experiment does not separately measure perceptual versus motor-learning sample complexity or vary physical prompts.It does not establish lower total pretraining cost, a universal data-scaling law, or driving-data efficiency.Source: §C.1 · p. 34

C.2 Grounding and Action Scaling

  • The grounding-and-action scaling analysis orders evaluations by increasing grounding-training exposure while keeping the initial backbone, action architecture, demonstrations, and training recipe fixed.Table 16 records the performance trajectory, but the supplied passage does not report its values.Source: §C.2 · p. 34, Table 16 · p. 34

C.3 Which Perceptual Capabilities Transfer?

The ablations show group-level complementarity in downstream transfer, while the OCR catalyst and embodied-foundation analyses examine how perceptual supervision, feature access, and adaptation may relate to action performance.

C.3 Which Perceptual Capabilities Transfer?

  • The full mixture leads both manipulation metrics and ties the best displayed driving error, while F alone is the weakest configuration.Adding A and D to F raises ID SR from 24.42% to 30.92% and OOD Avg SR from 7.28% to 29.28%, while driving error falls from 0.391 to 0.310 m; A and D change jointly, so their separate effects are unidentified.Source: §C.3 · pp. 34–35, Table 17 · p. 35

C.4 OCR as a Perceptual Catalyst: A Hypothesis

  • The OCR catalyst hypothesis is tested by removing only transcription loss while retaining protocol supervision, transcription tokens, conditioning, and matched training budgets.The proposed catalyst is that transcription loss improves non-text dense or tiny grounding and subsequent action transfer; Table 18 varies when OCR supervision is incorporated, with cold-start plus batch mixing corresponding to the full six-group configuration.Source: §C.4 · p. 36, Table 18 · p. 36

C.5 Output Representation and Generation Cost

  • 2.47 pp favors quantization in grounding Avg; the reported encoder and reinforcement-learning gaps are configuration-level comparisons, not isolated architectural effects.The reported textual speed is 0.25× the quantized configuration’s speed; token-per-box ratios quantify serialization rather than measured speedups, and Figure 12 indicates rising generation cost as more boxes are emitted, not real-time action performance.Source: §C.5 · pp. 36–37

D Embodied-Foundation Design: Mechanisms and System Roles

  • The embodied analyses characterize feature access and adaptation locally under a new action objective, while cross-model comparisons do not isolate architectural causes.Source: D Embodied-Foundation Design:… · p. 37

D.1 Architectural Facts and Scope of Comparison

  • Downstream capacity and optimization are fixed, but complete pretrained foundations vary, leaving encoder, connector, pretraining, positional treatment, and language-model scale as comparison differences.Source: §D.1 · p. 37

D.2 Gated Memory and Attention: Conditional Transfer Sensitivities

  • Action-feature accessibility depends on spatial readout, not gate values alone; recurrent value influence can decay with sequence separation under uniformly bounded gates.The conditional action sensitivity also depends jointly on the token path and downstream network, including possible amplification or cancellation; output gates modulate attention outputs.Source: §D.2 · pp. 37–38

D.3 Multi-level Visual Injection and Action Readout

  • Multi-level feature sensitivity is not prediction error, and extra branches preserve the single-interface model when they can be zeroed; practical gains depend on learning a compatible readout within the available budget.Shared projection and resampling couple adaptation across feature levels, while depth embeddings and separate action cross-attention can compensate; a matched-capacity comparison across action-data budgets would test the compatibility hypothesis.Source: §D.3 · p. 38

D.4 Designing Perceptual Foundations for System 1

The section proposes perception-native grounding as a foundation for the perception–action execution loop, distinct from planning and from a complete controller. It motivates this division while noting that an integrated real-time dual-system controller remains untested.

D.4 Designing Perceptual Foundations for System 1

  • The proposed division separates knowledge-intensive deliberation from rapid perception–action execution, while prior dual-system architectures motivate but do not establish an optimal universal split.RoboDual couples a generalist with a specialist policy, and GR00T N1 couples vision–language interpretation with a diffusion action module.Source: §D.4 · pp. 38–39
  • A perception-native foundation can supply precise, task-conditioned information to the execution loop, but the proposal is not an unadapted grounding VLM acting as a complete controller.System 1 denotes the functional perception–action execution loop; its module boundaries need not match another architecture’s labels.Source: §D.4 · p. 39
  • Reliable execution matters because stronger plans still depend on identifying interaction targets and executing feasible actions; the coding-agent analogy motivates evaluating the feedback loop, not a cross-domain empirical comparison.Code as Policies connects perception outputs with control APIs.Source: §D.4 · p. 39
  • Dense and tiny-object grounding can provide instance, location, and local-region distinctions for interaction, while OCR may complement grounding through local visual precision and semantic alignment.Grounding leaves dynamics, contact, and feedback control to the downstream policy; the stated pretraining priorities are suggestions, not demonstrated mechanisms.Source: §D.4 · p. 39
  • The proposed planner–execution allocation is prospective, and the paper calls for controlled tests of task success, distribution-shift robustness, and observation-to-action latency rather than claiming a real-time dual-system controller.The proposed tests would hold the planner and action expert fixed while changing the perceptual foundation, and vice versa.Source: §D.4 · p. 39

E Limitations and Future Work

The supplied benchmark passages report strong dense-scene results alongside persistent localization gaps, especially for tiny objects and strict box boundaries. The appendix also specifies how its comparisons and aggregate scores are reported.

E Limitations and Future Work

  • GroundingPI’s aggregate strength is not uniform: tiny-box localization, high-IoU boundaries, TotalText OCR, FSC147 visual prompting, and frontier-level GUI grounding retain gaps.The manipulation comparison controls the downstream recipe, but foundations differ in scale, architecture, and supervision; driving is evaluated open loop, and action-data efficiency only in manipulation.Source: E Limitations and Future Work · p. 39

F Comprehensive Grounding Benchmark Results

  • The appendix defines headline Avg as the mean of 11 independently computed capability scores and keeps benchmark-level results separate; dataset-family averages are unweighted and reported only when all components are available.It retains recorded threshold-specific and mean-overlap metrics, and does not infer missing recall or precision from F1.Source: F Comprehensive Grounding Benchmark… · p. 40

F.1 Common and Long-tailed Object Detection

  • 62.98 F1mIoU on COCO exceeds Rex-Omni, LocateAnything Hybrid, and Astra, while GroundingPI’s 56.02 on LVIS slightly trails SenseNova-Vision.COCO F1 at IoU 0.95 is 22.84; on LVIS it is 22.42, versus SenseNova-Vision’s 28.88.Source: §F.1 · p. 40

F.2 Dense and Tiny Object Detection

  • 74.53 F1mIoU on Dense200 exceeds SenseNova-Vision by 6.40 pp and Astra by 9.49 pp, while VisDrone’s 40.44 trails two baselines.On Dense200, GroundingPI records 27.85 F1 at IoU 0.95 versus Astra’s 12.64; on VisDrone, its 3.31 F1 at IoU 0.95 remains low, as do other results.Source: §F.2 · p. 40

F.3 Referring Object Detection

GroundingPI reports strong referring-box localization across HumanRef and RefCOCO-family benchmarks, though strict-boundary rankings do not always match aggregate rankings. Its family-level aggregate does not establish a universal lead.

F.3 Referring Object Detection

  • 88.56 F1mIoU and 76.47 F1 at IoU 0.95 on HumanRef exceed Astra’s 83.01 and 71.19, respectively.IoU 0.50 recall and precision are closely balanced at 93.10 and 93.93.Source: §F.3 · p. 41
  • 85.62/84.36 F1mIoU on RefCOCOg validation/test exceed Astra’s 74.98/78.91 and LocateAnything Hybrid’s 76.43/77.67, but DeepSeek-VL2-27B is stronger at IoU 0.95.DeepSeek-VL2-27B nearly matches the test aggregate at 84.07.Source: §F.3 · p. 41
  • 83.92 is GroundingPI’s reported mean across RefCOCO, RefCOCOg, and RefCOCO+ F1mIoU, while DeepSeek-VL2-27B leads the three complete-table aggregates.GroundingPI scores 85.79, 81.96, and 84.01, respectively; on RefCOCO+ it exceeds Qwen3-VL-4B, 84.01 to 73.36.Source: §F.3 · p. 42

F.4 Object Pointing

The pointing results evaluate point-in-mask correctness and show different strengths across referring, object, and spatial tasks. The reported comparisons distinguish target selection and precision from box-boundary quality and closed-loop manipulation.

F.4 Object Pointing

  • 88.79 F1@Point on HumanRef and 90.26/90.06 on RefCOCOg validation/test exceed Astra and selected grounding specialists.A point is scored correct when it falls inside a mask derived from the ground-truth box, so this measures point correctness rather than box-boundary quality.Source: §F.4 · pp. 42–43
  • 84.79 F1@Point on COCO and 79.51 on LVIS exceed Astra’s 82.17 and 77.14, although Astra has higher COCO recall.GroundingPI’s COCO precision is 85.38 versus Astra’s 78.14, while recall is 84.20 versus 86.56.Source: §F.4 · p. 43
  • 81.27 on Dense200 trails Astra’s 86.57, while GroundingPI’s 67.47 on VisDrone yields higher F1 than Astra’s 65.62.GroundingPI’s Dense200 precision is 85.91 versus Astra’s 83.74; VisDrone precision is 74.17.Source: §F.4 · pp. 43–44

F.5 Robot and Spatial Pointing

  • 76.00/75.00 on RefSpatial Location/Placement and 75.32 on Unseen are below Astra, while GroundingPI leads RoboSpatial Context, 73.77 to 65.69.The results suggest complementary strengths in instruction-conditioned placement and contextual localization, but do not directly measure closed-loop manipulation.Source: §F.5 · pp. 44–45

F.6 OCR

The OCR, GUI, layout, and visual-prompting evaluations show strong results on several benchmarks, alongside specific gaps in tight localization and professional interfaces.

F.6 OCR

  • On HierText and ICDAR2015, GroundingPI reaches 41.70/55.68 F1mIoU versus Astra’s 39.58/48.87; on SROIE it reaches 72.47 versus Astra’s 53.57.TotalText is less favorable: its 49.32 trails Astra and Rex-Omni, and the remaining gap concerns valid text-region predictions rather than output syntax alone.Source: §F.6 · pp. 45–46

F.7 GUI Grounding

  • ScreenSpot-V2 and OSWorld-G scores are 96.15 and 74.82, respectively, but Astra remains stronger overall on both benchmarks.On ScreenSpot-Pro, GroundingPI reaches 65.78 overall accuracy, below Astra’s 93.17; near-ceiling ScreenSpot-V2 performance does not mean broader GUI grounding is solved.Source: §F.7 · p. 47

F.8 Layout Grounding

  • On M6Doc, GroundingPI reaches 74.82 F1mIoU versus Astra’s 60.59; on DocLayNet, it scores 85.08 versus SenseNova-Vision’s 85.53.On DocLayNet, GroundingPI’s 43.47 at IoU 0.95 is below SenseNova-Vision’s 49.33; on M6Doc, gains extend across loose and strict overlap criteria.Source: §F.8 · p. 48

F.9 Visual Prompting

  • On FSC147, GroundingPI’s 57.99 F1mIoU slightly exceeds Rex-Omni’s 57.15 but trails Astra and SenseNova-Vision.Its 87.04 F1 at IoU 0.50 falls to 3.49 at IoU 0.95, exposing a boundary-precision limitation. On Dense200, it reaches 75.43 F1mIoU versus Astra’s 68.94, with the advantage reflecting tighter localization; on COCO and LVIS it reaches 82.62 and 78.32 F1mIoU, respectively. COCO scores should not be compared directly with category-prompted COCO because target information differs; unsupported exemplar interfaces are not zero-score measurements.Source: §F.9 · pp. 49, 50, 52

G Potential Applications

The paper outlines potential uses for grounding and presents qualitative examples of instance selection, spatial relations, point placement, and small-target localization. These examples illustrate task demands, not additional accuracy estimates.

G Potential Applications

  • The potential applications span industrial inspection, embodied-data annotation, dense retrieval and counting, document extraction, and GUI interaction.Examples include localizing defects or parts, finding unusual road obstacles, counting crowded objects, associating text with regions, and locating interface controls.Source: G Potential Applications · pp. 52–54

H.1 General Object Grounding

  • In a crowded classroom scene, broad category recognition must accompany precise localization of overlapping people, furniture, and smaller objects.The displayed Ours panel includes ground-truth completion, so its coverage is not evidence of unedited model recall.Source: §H.1 · pp. 54–55

H.2 Dense Object Grounding

  • Dense fruit grounding distinguishes individual, partially hidden contents from a salient basket, broad container region, or overlapping proposals.The example highlights the need to resolve both local appearance and the object level requested by the query.Source: §H.2 · p. 56

H.3 Referring Grounding and Complex Visual Configurations

  • The referring example selects the person whose hand is behind two others, requiring a relationship between partly occluded body parts and person-level regions.The illustrated comparison panels select the flanking people instead.Source: §H.3 · p. 57

H.4 Referring Point-in-Mask Grounding

  • For point-in-mask grounding, selecting the correct sofa instance and an exposed surface point matters; a nearby cushion or other sofa can be incorrect.A coarse box overlap alone would not ensure a correct point-in-mask prediction.Source: §H.4 · pp. 57–58

H.5 Dense Point Grounding

  • The aerial example contrasts spatially distributed instance-level points with repeated nearby points, amid small targets, perspective variation, and clutter.The query is not shown, so the example does not establish a target category.Source: §H.5 · p. 58

H.6 GUI Grounding

  • The GUI example distinguishes a small requested side-panel control from a prominent active cell, illustrating why localization of neighboring toolbar icons matters.The displayed Ours point aligns with the marked control, while the comparison point targets a central cell.Source: §H.6 · p. 61

H.7 OCR and Artistic Text Understanding

The OCR examples show text recognition across artistic, multilingual, and numerical layouts, while the visual-prompt example focuses on identifying individual objects in dense scenes.

H.7 OCR and Artistic Text Understanding

  • The OCR examples span curved or slanted artistic lettering, small print, multilingual packaging, and numerical fields, with text regions preserving spatial associations.The artistic-text display includes imperfect small-text transcriptions, so it is not evidence of error-free OCR; another example notes that text grouping can depend on annotation granularity.Source: §H.7 · pp. 61–63

H.8 Visual-Prompt Grounding

  • The pipe-opening example uses a visual exemplar to specify an individual opening, contrasting instance-level boxes with a broad group-level interpretation.Repeated dark interiors and narrow boundary gaps make the instances difficult to separate.Source: §H.8 · p. 64, Figure 22 · p. 64
Loading 2609.39601v1…