PaperlayerView exact revision on arXiv

Source-linked AI summary

PointRL: Learning Point-Level Vision-Language Grounding from Verifiable Annotation Evidence

Jingyang Su, Pu Cao, Xiuze Jin, Longyue Zhang, Qing Song, Lu Yang

arXiv:2608.25299v1cs.CV

TL;DR

Point-level grounding must accommodate multiple valid coordinates and enforce coverage, counts, and uniqueness for multi-instance instructions. PointRL uses heterogeneous annotations as hidden verifier evidence for structured reinforcement rewards, improving Qwen3.5-4B PointArena accuracy from 56.11% to 65.58% and gaining on evaluated external benchmarks.

  • Problem

    Point-level grounding has non-unique valid coordinates, while multi-instance instructions require target coverage, cardinality consistency, and duplicate suppression.

  • Method

    PointRL converts boxes, masks, and instance labels into instructions while retaining target supports, instance membership, and constraints as hidden verifier evidence for structured rewards.

  • Results

    PointArena overall accuracy for Qwen3.5-4B rises from 56.11% to 65.58%, with same-backbone gains also reported on RoboSpatial, BLINK, and Ref-Adv.

  • Takeaways & Limitations

    Verifiable point-level feedback may benefit spatial grounding in the evaluated settings, including tasks requiring collection structure and instruction-conditioned target selection.

Abstract

from arXiv · show

Vision-language models (VLMs) increasingly rely on point coordinates as a compact and executable interface for visual grounding in GUI interaction, robotic manipulation, and interactive visual systems. However, learning reliable pointing behavior remains difficult because the supervision space is inherently non-unique: many coordinates may be valid within the same target region, while multi-instance instructions require target coverage, count consistency, and duplicate suppression. This work presents PointRL, a verifiable reinforcement learning framework that learns point-level grounding from existing heterogeneous annotation evidence. PointRL converts bounding boxes, masks, and instance labels into pointing instructions, while retaining their target supports, instance membership, and set constraints as hidden verifier evidence, i.e., annotations kept outside the prompt and used by a deterministic checker to score predictions. The proposed reward evaluates parseability, point validity, instance coverage, cardinality consistency, and redundant or missing predictions. On PointArena, PointRL improves the overall accuracy of Qwen3.5-4B from 56.11% to 65.58%. Further evaluations on RoboSpatial, BLINK, and Ref-Adv show same-backbone gains on the evaluated external benchmarks, suggesting that verifiable point-level feedback may benefit spatial grounding in these settings.

I. INTRODUCTION

PointRL addresses the ambiguity and structural demands of point-level visual grounding by converting heterogeneous annotations into hidden verifier evidence and structured rewards. It improves same-backbone grounding on PointArena and shows gains on evaluated external benchmarks.

  • Point coordinates offer a compact, executable grounding interface for GUI agents, robotic manipulation, assistive perception, and interactive visual systems.
  • Many coordinates can validly represent one target region, while multi-instance instructions additionally require complete coverage, correct cardinality, and distinct predictions.
  • PointRL converts boxes, masks, and instance labels into pointing instructions while preserving target supports, instance membership, and constraints as hidden verifier evidence.
  • Its structured reward evaluates parseability, point validity, localization, instance coverage, cardinality consistency, and redundant or missing predictions.
  • 56.11% to 65.58%: PointRL improves Qwen3.5-4B overall accuracy on PointArena and shows same-backbone gains on RoboSpatial, BLINK, and Ref-Adv.

B. Language-Guided Pointing

The paper frames language-guided pointing as an executable interface for grounding, then proposes verifier rewards that evaluate unordered point collections rather than single fixed coordinates. This design addresses valid-point ambiguity and multi-instance structure.

  • Point prediction provides image coordinates that can directly serve as clicks, manipulation targets, or prompts for downstream segmentation.
  • PointArena evaluates pointing across spatial, affordance, counting, steerability, and reasoning-oriented instructions, while pointing also functions as an action interface in robotics and GUI grounding.
  • Existing visual verifiable rewards often target task correctness or region quality, but point grounding requires multiple valid coordinates, coverage, count consistency, and duplicate suppression.
  • A pointing instance supplies an image, instruction, and hidden verifier evidence, while the model receives only the image and instruction and returns points.
  • Verification treats point collections as unordered, preserving repeated coordinates so missing, extra, and duplicated predictions can be penalized.
  • Correct responses must match target supports and satisfy constraints such as distinct-target coverage, cardinality, uniqueness, and relations.

B. Dataset Construction

PointRL constructs verifiable pointing samples by normalizing heterogeneous annotations into entities and evidence, then applying rules to generate instructions with fixed hidden targets and constraints. The accepted dataset combines geometric and instruction-conditioned rule families.

  • The construction pipeline normalizes raw annotations, collects evidence for target and reference entities, and instantiates instructions with hidden verifier evidence V = (T, C).
  • Each normalized entity stores an identifier, optional semantic label, support geometry, and source metadata, while object evidence records labels, relations, counts, supports, and functional roles.
  • Rule-based construction selects category, attribute, relation, counting, region, or functional-role patterns and accepts samples only when the answer reduces to a verified target set.
  • A Qwen-based refiner rewrites instruction surface forms while deterministic checks preserve target supports, target ambiguity constraints, cardinality, and alignment with hidden evidence.
  • Valid targets retain support regions for hit checking and soft localization, while multi-target samples store target sets and cardinality constraints for unordered matching.
  • The construction yields 1,647 training samples and a 200-sample held-out diagnostic split, with relation and cue-based families comprising 48.0% together.

IV. POINTRL: REINFORCEMENT LEARNING FOR POINTING

PointRL parses textual responses into unordered point collections and scores them against hidden verifier targets. Its reward combines graded localization with one-to-one matching, validity checks, and penalties for missing or extra predictions.

  • Point prediction and parsing: The VLM response is deterministically parsed into a point collection P, with malformed, empty, conflicting, or out-of-bounds answers rejected.Parseable responses require one normalized coordinate list containing at least one valid numeric tuple.
  • Localization scoring: PointRL converts prediction-to-support distance into a smooth localization score, giving near misses graded feedback instead of binary credit.Inside-region points receive zero distance, while outside points are measured against the nearest support point.
  • Target supports: Each verifier target stores a set-valued support, allowing any point inside a box- or mask-derived region while treating annotated points as singleton supports.This preserves multiple valid coordinates for region supervision.
  • Localization scoring: Outside-region scores are capped by support reliability so near misses retain feedback without receiving the highest localization credit.Mask or annotated-region supports use the highest cap, box supports an intermediate cap, and weak supports the lowest cap.
  • Prediction-target matching: Hungarian matching assigns unordered predictions to targets one-to-one, separating matched pairs from unmatched predictions and missed targets.The resulting score matrix supports differing prediction and target counts without double counting.

C. Hierarchical Pointing Reward

The hierarchical reward combines local matched-point quality with global collection consistency and instruction-specific guards. This design addresses singleton and multi-target failures, including missed targets, wrong counts, duplicates, and cue copying.

  • Reward hierarchy: PointRL builds the response reward in three stages: local matching quality, collection-level consistency, and instruction-conditioned guards.The same formulation applies to singleton and multi-target pointing.
  • Local reward: Local reward averages matched credit from prediction-side and target-side views, so extra predictions and missed targets reduce different normalized scores.Unmatched predictions remain in the prediction denominator, while missed targets remain in the coverage denominator.
  • Global reward: Under-counting is penalized more strongly than over-counting because the chosen coefficients satisfy γu > γo.This prioritizes avoiding missed required targets.
  • Global reward: Global reward multiplies target coverage, count consistency, and duplicate suppression to penalize incomplete, overlong, or collapsed point collections.These terms enforce set structure beyond individual localization quality.
  • Instruction-conditioned guards: Instruction guards reject invalid constraints and suppress cue copying when predictions stay near an auxiliary anchor without entering any target support.An optional direction-shaping term is subordinate to the main localization and consistency rewards.
  • Instruction-conditioned guards: The format gate removes all reward from unparsable responses after the guarded reward is computed.The core verifier reward is distinct from additional fixed training safeguards used for the scalar GRPO reward.

D. Training Details

PointRL trains VLMs with serialized point collections scored by hidden-verifier rewards, then updates the policy using group-normalized advantages and clipped GRPO.

  • D. Training Details: PointRL serializes category-specific JSON responses into point collections evaluated by hidden verifier evidence.The responses are rescaled to image pixels before verification.
  • D. Training Details: The scalar reward includes safeguards for severe count mismatch, repeated points, and excessive unmatched predictions.These cases are capped at fixed floors held constant across reward configurations.
  • D. Training Details: Rewards are normalized within each response group to compute sample advantages using the group standard deviation and a numerical-stability constant.The normalization produces the advantage applied during policy optimization.
  • D. Training Details: The model is updated with a clipped GRPO objective based on the policy ratio relative to the sampling policy.The reference-policy KL term is retained in the general objective but disabled with β = 0.

V. EXPERIMENTS

The experiments test whether verifier-based point-level training improves a base VLM on PointArena, how reward configurations behave, and whether gains transfer to external benchmarks without benchmark-specific fine-tuning.

  • V. EXPERIMENTS: The evaluation is organized around same-backbone improvement, reward-configuration behavior, and external-benchmark gains without benchmark-specific fine-tuning.These questions define the experimental scope for PointArena and external spatial grounding benchmarks.

A. Experimental Setup

The setup evaluates PointRL on PointArena and external benchmarks using binary hit-rate metrics, controlled same-backbone comparisons, fixed implementation settings, and cautious interpretation of single-checkpoint differences.

  • A. Experimental Setup: Point-Bench covers affordance, counting, reasoning, spatial, and steerable pointing tasks, with 982 samples in its test set.PointRL uses 1,647 constructed training samples and a separate 200-sample diagnostic validation split.
  • A. Experimental Setup: The reported hard hit rate requires valid points inside target masks and correct counts for counting or multi-target collections.Training instead uses a denser hidden-verifier reward.
  • A. Experimental Setup: Controlled comparisons pair Qwen3.5-2B and Qwen3.5-4B base models with their PointRL-trained counterparts under the same evaluation protocol.Other reported models serve as contextual references rather than controlled evidence.
  • A. Experimental Setup: All variants use Qwen3.5 backbones and category-specific JSON serialization normalized to point collections before verification.The main base model is Qwen3.5-4B, with Qwen3.5-2B used for a smaller-scale run.
  • A. Experimental Setup: Small differences are interpreted cautiously because each configuration is evaluated from a single checkpoint and statistical significance is not claimed for several comparison types.The reported metrics are binary hit rates without per-example agreement statistics.

B. Main Results

PointRL improves same-backbone Point-Bench performance, with the largest 4B gains on counting, reasoning, and steerable tasks and qualitative examples showing more instruction-relevant coverage.

  • B. Main Results: 65.58% overall accuracy improves from 56.11% for Qwen3.5-4B, an observed gain of 9.47 percentage points.Under the same reward settings, Qwen3.5-2B improves from 25.25% to 56.82%.
  • B. Main Results: 13.26, 11.92, and 10.50 points are the largest Qwen3.5-4B gains on counting, reasoning, and steerable tasks, respectively.Affordance improves by 9.59 points, while spatial improves by 2.05 points from a 69.23% baseline.
  • B. Main Results: The strongest gains occur on tasks requiring collection structure or instruction-conditioned target selection.These tasks require deciding both how many targets to return and which constraint-bound target set is valid.
  • B. Main Results: Qualitative examples show PointRL moving predictions toward instruction-relevant regions and improving target coverage in multi-target cases.The examples are consistent with stronger gains on counting, reasoning, and steerable categories.

C. Reward Ablation

The full PointRL reward is the strongest tested configuration, while partial rewards provide progressively useful but incomplete signals for multi-target pointing. The comparison is configuration-level evidence rather than an isolation of individual reward terms.

  • 65.58% is the highest observed overall accuracy for the full reward, 9.47 points above the base model and at least 3.16 points above partial variants.The comparison uses a shared base model and evaluation protocol.
  • Figure 6 compares six reward configurations on multi-target examples, with red points marking predicted point collections.Partial rewards can leave missing, collapsed, or duplicated points, whereas the full reward better covers the target set in the shown examples.
  • 62.42% overall accuracy after adding target coverage represents a further 2.24-point gain over localization-only reward.The added coverage mainly improves reasoning, spatial, and steerable tasks.
  • Cue-cap configurations improve over the base model, but progress shaping slightly increases affordance while reducing steerable performance and overall accuracy relative to cue cap alone.
  • The table does not isolate count consistency or duplicate suppression, so their marginal contributions require additional controlled variants and repeated runs.

D. External Benchmarks and Distractor-Stratified Analysis

PointRL yields positive same-backbone gains across the evaluated external benchmarks and output protocols, with magnitude varying by benchmark and split. Ref-Adv results indicate bounded transfer to target selection under box output rather than direct point-output performance.

  • Ref-Adv evaluates target selection through the benchmark-required box-output protocol, while RoboSpatial and BLINK test spatial grounding directly.
  • 7.38, 7.23, 13.34, and 7.00 percentage-point gains occur on RoboSpatial context, configuration, compatibility, and BLINK, respectively.These are observed gains over the Qwen3.5-4B baseline under the evaluated external metrics.
  • Figures 7 and 8 provide qualitative examples for Ref-Adv and RoboSpatial-Home, including Ref-Adv groups with 2–3, 4–6, and ≥7 distractors.
  • The compatibility split has the largest same-backbone improvement, yet PointRL remains below methods trained directly with RoboSpatial data on absolute compatibility score.The BLINK gain occurs without BLINK-specific tuning, supporting transfer beyond the constructed PointRL-Data distribution.
  • Ref-Adv scores improve by 2.01, 3.33, and 1.93 points across the three IoU thresholds, and by 1.29, 3.49, and 2.32 points across distractor strata.The distractor strata are 2–3, 4–6, and ≥7 distractors.
  • The Ref-Adv pattern is compatible with improved target selection rather than a formatting advantage because PointRL leaves the output protocol unchanged.The improvement is reported as more reliable intended-object selection amid competing distractors.
  • External evaluations show positive same-backbone gains under both point- and box-output protocols, but their magnitude varies by benchmark and split.

VI. CONCLUSION

PointRL learns point-level visual grounding from heterogeneous annotations by combining hidden verifier evidence with structured rewards over point collections. It improves grounding performance in the evaluated settings, including a 9.47-point gain over the Qwen3.5-4B baseline on PointArena.

  • PointRL preserves heterogeneous grounding annotations as hidden verifier evidence while converting boxes, masks, and instance labels into pointing instructions.This retains the original annotation information for reward computation rather than exposing it in the prompt.
  • PointRL uses a structured reward over unordered point collections to support single-target and multi-target grounding.The reward combines parseability, soft localization, target coverage, count consistency, duplicate suppression, and constraint checks.
  • 9.47-point gain over the Qwen3.5-4B baseline was observed on PointArena under the evaluated protocol.
  • Configuration-level ablations suggest that local localization feedback and collection-level structural constraints are useful in combination in the tested settings.
  • Bounded same-backbone gains were observed on RoboSpatial, BLINK, and Ref-Adv under the evaluated protocols.These results suggest that heterogeneous grounding annotations can serve as useful hidden verifier evidence for improving VLM pointing ability in the evaluated settings.
Loading 2608.25299v1…