Source-linked AI summary
Pointing-VLA: Typed Spatial Grounding Interfaces for Vision-Language-Action Manipulation
Xiwen Chen, Zelin Li, Zhiruo Zhou, Huiming Chen, Chenwei Wang, Xiaojun Zhu
TL;DR
VLA systems often lack a robust, inspectable interface for converting multimodal spatial reasoning into robot execution. Pointing-VLA uses typed hidden-state readouts and stage-aligned execution contracts, achieving strong benchmark, transfer, runtime, and real-robot results. The paper concludes that geometry-appropriate readouts provide an efficient interface between embodied reasoning and execution.
Problem
VLA models commonly expose spatial grounding through brittle text coordinates or opaque action tokens, limiting direct and inspectable geometry for robot execution.
Method
Pointing-VLA reads multimodal hidden states with separate point, OFG heatmap, and visual-trajectory heads, using OFG for PICK and Pointing for PLACE.
Results
Pointing-VLA achieves SOTA Bridge/WidowX performance, averaging 72.9% across four tasks without Bridge-specific finetuning, while showing complementary geometry strengths across evaluations.
Takeaways & Limitations
Typed spatial readouts provide an efficient, inspectable interface connecting embodied VLA reasoning to robot execution.
Takeaways & Limitations
The fixed pick/place scaffold is a controlled deployment setting, and task-dependent variation identifies closed-loop geometry-aware execution as a next leverage point.
Abstract
from arXiv · showhide
Vision-language-action (VLA) models often expose spatial grounding through autoregressive text coordinates or opaque action tokens, creating brittle interfaces between multimodal reasoning and robot execution. We present Pointing-VLA, a typed hidden-state spatial readout built on Embodied-R1. Geometry-specific heads predict normalized points, object-functional grounding (OFG) heatmaps, and visual trajectories without serializing geometry as text. For the evaluated Bridge/WidowX and physical pick-place deployments, an explicit execution contract assigns PICK to source-conditioned OFG and PLACE to Pointing, providing direct stage-aligned spatial targets. Pointing-VLA achieves SOTA performance on Bridge/WidowX, averaging 72.9\% across the evaluated four-task set without Bridge-specific finetuning under collision-enabled CuRobo execution. Pointing and OFG show complementary strengths across native and cross-dataset evaluations. The OFG/contact readout transfers to NORA-1.5, preserving or improving success while reducing recorded controller time by more than 20$\times$; typed heads are also 6.68--6.90$\times$ faster than Embodied-R1 text decoding on a shared external suite. When integrated as spatial guidance for a $π_{0.5}$ action policy, Pointing-VLA raises autonomous real-robot success from 52.7\% to 80.7\% across three visual contexts. These results establish typed spatial readouts as an efficient, inspectable interface between embodied reasoning and robot execution.
Introduction
Pointing-VLA reframes embodied grounding as typed spatial interface prediction, replacing brittle text-coordinate serialization with geometry-specific readouts aligned to robot execution. Its evaluations span native grounding, transfer, runtime, benchmark, and physical deployment.
- Motivation: Text-coordinate interfaces can fail during generation or produce parseable coordinates outside the target region.Parsing adds an execution dependency without guaranteeing correct spatial grounding.
- Core idea: Typed interface prediction distinguishes sparse points, dense functional regions, and temporally structured trajectories instead of collapsing them into one distribution.The paper argues these geometries are related but not identical execution targets.
- Core idea: Pointing-VLA reads spatial intent from multimodal hidden states through separate point, OFG, and visual-trajectory decoders.The heads emit geometry directly for robot-side execution rather than serializing coordinates as text.
- Deployment: The primary pick-place contract assigns source-conditioned OFG to PICK and Pointing to PLACE, aligning each execution stage with its required geometry.This creates deterministic, stage-aligned spatial targets for downstream wrappers, planners, and executors.
- Evaluation: Cross-dataset transfer, runtime evaluation, and physical deployment extend validation beyond native grounding.The results report complementary Pointing and OFG geometries, more than 20× shorter recorded controller time on NORA-1.5, and real-robot execution.
Related Work
Prior work improves generalist robot policies, spatial manipulation representations, and action-sequence generation, while Pointing-VLA contributes typed spatial targets before low-level action generation.
- Generalist robot policies: Generalist VLA and robot-policy research adapts pretrained models to robot action spaces and improves policy learning, action parameterization, and inference throughput.The cited line includes embodied multimodal models, Robotics Transformers, cross-embodiment datasets, and open robot policies.
- Spatial manipulation representations: Neuroscience and robot-learning work preserve action-facing spatial structure through intention maps, affordance candidates, spatial feature maps, semantic-spatial pathways, and voxel actions.Examples include Transporter Networks, CLIPort, and PerAct for language-conditioned manipulation.
- Relation to action policies: Pointing-VLA complements Diffusion Policy and ACT by exposing typed spatial targets before low-level action generation.In evaluated pick-place systems, functional contact is decoded for PICK and a compact target point for PLACE.
Method
Pointing-VLA decodes geometry-specific robot targets from shared multimodal hidden states through separate spatial heads and structured execution contracts. Its staged training and refinement procedures specialize these heads for points, affordance heatmaps, and visual trajectories without serializing geometry as text.
- Hidden-State Spatial Readout: Pointing-VLA decodes geometry-specific spatial outputs from shared Embodied-R1 hidden states through separate point, OFG, and VTG heads.The architecture reuses a shared backbone and adapter while retaining specialist-specific LoRA, learned queries, and decoder state.
- VTG Head: VTG uses learned temporal queries over the full multimodal sequence to produce eight normalized image-space waypoints describing a visual trace.The reported trajectory metrics are RMSE, ADE, and FDE; the waypoints are not low-level robot actions.
- Pointing Head: Pointing uses a learned pointer query to select grounded evidence for one image location, emitting normalized coordinates directly without coordinate-text generation or parsing.The output is evaluated with point-in-bbox and point-in-mask metrics.
- OFG Head: OFG preserves spatial extent by FiLM-modulating a visual feature map before heatmap decoding, with the normalized peak serving as the functional contact.This readout retains dense part-level evidence rather than collapsing the source to one token.
- Training: Stage-wise training separates geometry warm-up, LoRA specialization, and task continuation, updating only the modules required at each stage.Frozen and trainable modules are explicitly marked throughout the procedure.
- Structured PICK/PLACE Contracts: The pick-place scaffold constructs slot-specific queries from the instruction, source description, and goal description, assigning OFG to PICK and Pointing to PLACE.An external wrapper converts image-space outputs into robot-frame targets, while the learned heads remain image-space predictors.
- Training: Training combines geometry-specific losses for points, regions, and trajectories, while disabling language-modeling and legacy reconstruction terms.Source-conditioned OFG specialization jointly updates the OFG LoRA, shared adapter body, OFG query, and heatmap decoder with dense reconstruction and distractor-aware ranking.
- Training: The final pointing model applies group-normalized refinement with continuous spatial-head optimization using supervised SmoothL1 loss, policy refinement, reference KL, and variance regularization.The policy center is the predicted point and the learned scale is parameterized as σ = exp(s).
Experiments
The experiments evaluate typed spatial readouts across native grounding, cross-dataset transfer, runtime, simulated manipulation, and real-robot deployment. Results show geometry-specialized heads and fixed execution contracts improve task-aligned grounding and execution efficiency.
- Native spatial grounding: Pointing-VLA matches Embodied-R1 on sparse referring-point grounding at 64.3%, while OFG improves full Part-Affordance-2K accuracy from 40.9% to 57.3%.The 16.4-point gain is attributed to retaining spatial extent needed for functional contact prediction.
- Native spatial grounding: VTG records 0.1042 RMSE, 0.1368 ADE, and 0.1493 FDE on 300 VABench-V examples.These metrics evaluate ordered waypoint sequences and terminal targets in normalized image coordinates.
- Fixed-contract validation: The fixed OFG-PICK/Pointing-PLACE contract completes 70/96 Bridge/WidowX episodes (72.9%), outperforming the strongest alternative by 14.6 percentage points.All Stack episodes and 22/24 Eggplant episodes are completed under collision-enabled CuRobo execution.
- Fixed-contract validation: Source-conditioned OFG completes 43/48 frozen multicolor Stack episodes versus 40/48 for Attention PICK, while selecting the instructed source in all 48 cases.Both conditions use identical Pointing PLACE targets.
- Cross-dataset geometry: Pointing is strongest on RefCOCO and RoboAff, whereas OFG raises full Part-Affordance-2K PIM from 40.9% with Embodied-R1 text generation to 57.3%.The crossover supports expert-specific output geometries rather than one serialized coordinate interface.
- Runtime and transfer: Typed geometric readouts run 6.68–6.90× faster than autoregressive text decoding, while transferred OFG/contact on frozen NORA-1.5 improves laid-vertical success from 89.0% to 95.0% and cuts controller time by more than 20×.The runtime comparison uses a shared external protocol, and the transfer result uses a shared wrapper.
- Real-robot deployment: Across three visual contexts, typed spatial guidance reduces pre-lift grasp failures from 47 to 16 and tray-arrival failures from 13 to 4.Post-grasp transfer remains the primary residual bottleneck.
Discussion and Conclusion
Pointing-VLA presents geometry-appropriate typed readouts as an efficient and inspectable interface between embodied reasoning and robot execution. The fixed pick/place scaffold is controlled, while task-dependent variation motivates closed-loop geometry-aware execution and broader transfer.
- Discussion and Conclusion: Typed spatial readouts expose points, heatmaps, and trajectories directly from shared multimodal states, aligning each geometry with its robot-facing execution stage.The interface treats referring points, functional regions, and waypoint traces as distinct targets.
- Discussion and Conclusion: The fixed pick/place scaffold provides a controlled deployment setting, while task-dependent variation identifies closed-loop geometry-aware execution as the next leverage point.Future work extends the interface with visual correction, broader structured task decompositions, and cross-robot transfer.