PaperlayerView exact revision on arXiv

Source-linked AI summary

Decoding Geometry, Not Tokens: Efficient VLMs for Robot Affordance and Trajectory Prediction

Akila Ayanthi, Darshana Priyasad, Tharindu Fernando, Dimity Miller, Sridha Sridharan, Clinton Fookes, Peyman Moghadam

arXiv:2610.39416v1cs.ROcs.CV

TL;DR

Robotic affordance and trajectory prediction require precise continuous geometry, whereas standard VLMs generate tokenized language. The paper attaches a lightweight geometry head to a frozen compact VLM and directly regresses boxes and waypoints. It reports improved prediction across both tasks, with its 2B model approaching RoboBrain on trajectory prediction at higher throughput.

  • Problem

    Standard VLMs generate tokenized coordinates rather than continuous geometric outputs for manipulation, which requires predictions of where to interact and how to move.

    Source: §1 · pp. 1–2

  • Method

    A lightweight geometry-compatible head reads intermediate features from a frozen VLM and directly regresses affordance boxes and trajectory waypoints.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Across both model families, the approach reduces trajectory DFD by over 68% and improves affordance mAP by up to 8.7× over corresponding vanilla small VLM baselines; the 2B model approaches RoboBrain on trajectory prediction while running 3.8× faster.

    Source: §4.2 · p. 5, §5 · p. 8

  • Takeaways & Limitations

    Compact VLMs already encode useful spatial information, and a geometry-compatible interface can exploit it while preserving compact-model efficiency.

    Source: §1 · p. 2

    The evaluation is limited to instruction-conditioned 2D geometric prediction prior to execution, leaving 3D settings, closed-loop robotics, and downstream manipulation success for future work.

    Source: §5 · p. 8

Abstract

from arXiv · show

Predicting where to grasp and how to move is fundamentally a geometry problem. Vision-language models treat it as language generation instead. They tokenize spatial coordinates into text and decode them autoregressively. This mechanism ignores spatial structure and truncates continuous precision. Errors compound across coordinate tokens during decoding. In this paper, we show that compact off-the-shelf VLMs with 0.5-2B parameters already encode rich spatial information in their intermediate hidden states. Autoregressive decoding is the dominant bottleneck for trajectory prediction, while affordance prediction depends additionally on feature depth and decoder architecture. We expose the latent spatial information with a lightweight, geometry-compatible prediction head that attaches to a frozen backbone. It reads features from an intermediate hidden layer, conditions a learned query on the task instruction and refines it via cross-attention over the visual tokens to regress affordance boxes and trajectory waypoints directly as continuous values, without modifying the backbone. We evaluate on ShareRobot-Bench and AGD20K. Our 2B variant reduces trajectory error by over 60% and improves ShareRobot-Bench affordance mAP by 85% relative to the autoregressive baseline. It approaches RoboBrain on trajectory prediction (0.251 vs 0.191 DFD) at 3.8x higher throughput. These results demonstrate a favorable accuracy-efficiency trade-off for instruction-conditioned 2D geometric prediction.

1. Introduction

Robotic manipulation needs instruction-conditioned predictions of where to interact and how to move, but standard VLMs generate tokenized language rather than continuous geometry. This paper uses a lightweight geometric prediction interface to tap spatial information in compact VLMs, reporting gains across affordance and trajectory prediction.

1. Introduction

  • The paper asks whether compact VLMs can provide accurate instruction-conditioned affordance and trajectory predictions with efficient inference.These outputs describe where to interact with a scene and how to move there.Source: §1 · p. 2
  • Tokenized coordinate generation mismatches geometric prediction: it lacks geometric-distance penalties, quantizes continuous outputs, and compounds errors across coordinate tokens.Larger VLMs can improve robotics performance but require more compute and have lower inference speeds.Source: §1 · p. 2
  • A lightweight geometry-compatible decoder directly regresses affordance boxes and trajectory waypoints from a frozen VLM.The paper identifies a decoding-interface mismatch, rather than proposing the DETR architecture itself as its contribution.Source: §1 · p. 2
  • A geometry-compatible interface substantially improves affordance and trajectory prediction while preserving compact-VLM efficiency.Direct continuous prediction is the primary source of trajectory improvement, whereas affordance prediction depends on the prediction interface, representation depth, and decoder architecture.Source: §1 · p. 2
  • Compact VLMs already encode useful spatial information across their representation hierarchy, although accessibility varies by task and model depth.Source: §1 · p. 2

2. Related Work

Compact VLMs offer an alternative to deploying very large models, but their language-generation training does not directly optimize geometric reasoning. This work uses frozen compact VLMs with lightweight geometry-aware decoders for affordance and trajectory prediction.

2.1. Vision-Language Models

  • VLMs trained for next-token prediction are optimized for language generation, and spatial tasks often use full fine-tuning or parameter-efficient adaptation to learn task-specific geometric representations.This work instead asks whether frozen compact VLM representations already contain sufficient geometric information for affordance and trajectory prediction.Source: §2.1 · p. 2

2.2. VLMs for Embodied Tasks

  • Embodied VLMs support robotics tasks including representation learning, planning, and feedback, while some embodied models rely on extensive backbone adaptation and large datasets.Large VLM backbones can be challenging to deploy on resource-constrained platforms.Source: §2.2 · p. 2
  • Unlike approaches that design new VLA architectures or heavily fine-tune backbones, this work uses compact off-the-shelf VLMs as frozen perception modules with lightweight geometry-aware decoders.Source: §2.2 · p. 3

3. Methodology

The method treats a compact VLM as a frozen visual-linguistic encoder and learns a lightweight head to regress continuous geometric outputs. Separate affordance and trajectory heads use task-specific losses while leaving the backbone unchanged.

3. Methodology

  • A frozen VLM and lightweight geometry head map intermediate hidden features to continuous spatial outputs instead of generating coordinate tokens autoregressively.The method targets a decoding-strategy bottleneck while keeping the backbone unchanged.Source: §3 · p. 3, §3.1.2 · p. 3

3.1. Model Architecture

  • The architecture pairs a frozen shared visual-linguistic encoder with a lightweight DETR-style decoder and task-specific FFNs, with all learned parameters in the prediction head.Source: §3.1 · p. 3

3.1.1. Vision-Language Model (VLM) Backbone

  • The VLM jointly contextualizes visual patch features and instruction-token embeddings, and no backbone weights are modified.Improvements therefore arise from the learned geometric prediction interface operating on frozen representations.Source: §3.1.1 · p. 3

3.1.2. Geometry Prediction Head

  • A DETR-style decoder refines an instruction-conditioned learned query through cross-attention over visual memory, then separate FFNs regress boxes and waypoints.The query uses the final text-token hidden state, and the two task heads maintain independent weights.Source: §3.1.2 · p. 3, Figure 2 · p. 4

3.2. Training

  • Training signals flow exclusively through the geometry prediction head, while the VLM backbone remains frozen.Source: §3.2 · p. 4

3.2.1. Affordance Loss

  • Affordance boxes are trained with a weighted combination of Smooth L1 coordinate regression and Generalized IoU loss.The Smooth L1 term applies element-wise to box corners.Source: §3.2.1 · p. 4

3.2.2. Trajectory Loss

  • Trajectory training combines Smooth L1 over all waypoints with additional penalties on the first and last waypoints.The endpoint terms upweight errors at the trajectory’s contact and termination points.Source: §3.2.2 · p. 4

4. Experimental Analysis

Across compact Qwen2-VL-2B and LLaVA-OV-0.5B backbones, geometry heads improve affordance and trajectory prediction while retaining strong inference throughput. Ablations show that continuous prediction is central to trajectory gains, while affordance accuracy also varies with representation depth and decoder design.

4.2.1. Trajectory Prediction

  • Over 60% lower DFD across both backbones, with proportional HD and RMSE reductions, accompanies geometry-head trajectory prediction; Qwen and LLaVA reach 0.251 and 0.225 DFD versus RoboBrain’s 0.191, at up to 3.8× faster inference.LoRA fine-tuning only marginally reduces DFD and remains substantially below the geometry-head variants.Source: §4.2.1 · p. 5

4.2. Trajectory and Affordance Prediction Results

  • Over 68% lower trajectory DFD and up to 8.7× higher affordance mAP than corresponding vanilla small-VLM baselines accompany geometry heads, which run at 7.9–21 FPS on H100.Table 1 compares accuracy and inference efficiency across models and baselines.Source: §4.2 · p. 5, Table 1 · p. 6

4.2.2. Affordance Prediction

  • Qwen2-VL-2B affordance mAP rises 1.7× (7.30 to 12.37 ± 1.16), while LLaVA-OneVision-0.5B rises from 1.09 to 9.53 ± 1.44 with geometry heads.Gains come with equal or better runtime than corresponding token heads; LoRA does not consistently improve over frozen baselines.Source: §4.2.2 · p. 6

4.2.3. Inference Efficiency

  • Geometry-head models generally achieve the highest throughput across both tasks; on RTX 3090, RoboBrain runs at 1.2/0.8 FPS while achieving the strongest overall performance.Inference stops at a selected intermediate layer, reducing active computation and FLOPs; the selected layer and skipped computation are backbone- and task-dependent.Source: §4.2.3 · p. 6

4.3. Disentangling Prediction Interface, Representation Depth and Decoder Architecture

  • For trajectory prediction, final-layer continuous MLP regression reduces DFD from 0.798 to 0.297±0.002 for Qwen and from 0.777 to 0.298±0.002 for LLaVA, accounting for 91.6% and 86.8% of full-model improvement.For Qwen affordance prediction, moving the MLP to an intermediate layer recovers 40.6% of the full gain, and DETR adds further improvement; the prediction interface alone does not explain the full affordance gains.Source: §4.3 · pp. 6–7, Table 2 · p. 7

4.4. Accessibility of Spatial Information Across VLM Representations

  • Pre-LLM features already support strong geometric prediction; trajectory DFD is substantially below autoregressive baselines at every evaluated hidden layer, while affordance gains vary substantially with depth.The layer sweep selects different layers by task and backbone: trajectory l = 13 for Qwen and l = 20 for LLaVA; affordance l = 12 and l = 23, respectively.Source: §4.4 · pp. 7–8, Figure 3 · p. 7
  • The preferred decoder-query representation depends on backbone and task, while explicit start/endpoint supervision consistently improves trajectory accuracy and reduces collapse toward a dataset-level mean.Source: §4.4 · p. 8

4.5. Qualitative Results

  • Qualitative comparisons show fragmented, spatially inconsistent baseline waypoints versus coherent geometry-head trajectories; for affordances, the head produces tighter, more coherent predictions than vanilla VLMs.Vanilla affordance predictions often roughly localize the relevant interaction region.Source: §4.5 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8

5. Conclusion

The geometry-compatible head improves affordance and trajectory prediction while preserving compact-VLM efficiency; the 2B model approaches RoboBrain on trajectory prediction at higher throughput. The evaluation remains limited to instruction-conditioned 2D prediction before execution.

5. Conclusion

  • The 2B model approaches RoboBrain on trajectory prediction while running 3.8× faster (21 vs. 5.5 FPS) and using less computation (1.2 vs. 31 TFLOPs).The geometry-compatible interface improves both tasks while preserving compact-VLM efficiency.Source: §5 · p. 8
  • Compact 0.5B–2B VLMs encode useful spatial information across their representation hierarchies, but accessibility varies by task and model depth.Trajectory gains primarily come from direct continuous prediction, whereas affordance prediction depends more on interface, representation depth, and decoder architecture.Source: §5 · p. 8
  • The evaluation covers instruction-conditioned 2D geometric prediction before execution, leaving 3D and closed-loop settings and downstream manipulation success for future evaluation.The authors also identify separate task heads and the storage needed for the backbone up to the selected early-exit layer as limitations.Source: §5 · p. 8

Decoding Geometry, Not Tokens: Efficient VLMs for Robot Affordance and Trajectory

The supplied passage labels supplementary material on prediction.

Decoding Geometry, Not Tokens: Efficient VLMs for Robot Affordance and Trajectory

A. Training

Affordance training combines coordinate regression and overlap losses, while trajectory training applies coordinate loss across waypoints and emphasizes endpoints. The appendix also describes AGD20K box construction and the inference-efficiency profiling protocol.

A.1. Affordance Loss

  • Affordance training combines Smooth L1 coordinate regression with Generalized IoU loss, using β = 0.05 to increase sensitivity to small coordinate errors.The overlap loss uses the smallest axis-aligned box enclosing the predicted and ground-truth boxes.Source: §A.1 · p. 11

A.2. Trajectory Loss

  • Trajectory training applies Smooth L1 uniformly across all waypoints and adds explicit penalties for the first and last waypoints.The endpoint penalties upweight trajectory start and end relative to intermediate waypoints; β = 0.05 increases sensitivity to small coordinate errors.Source: §A.2 · p. 11

B. AGD20K Affordance Box Construction

  • AGD20K affordance boxes are generated from evaluation-set binary masks using connected-component analysis and a 0.7 mask-to-box threshold.The resulting boxes serve as ground-truth labels and use the main paper’s bounding-box evaluation protocol.Source: B. AGD20K Affordance Box Construction · p. 11

C. Inference Efficiency Protocol

  • Inference efficiency is profiled at batch size 1 on H100 and RTX 3090 GPUs, with FPS measured after warm-up and FLOPs computed over 10 samples.Model loading is excluded; early-exit FLOPs include computation through l* and the geometric head but exclude later layers.Source: C. Inference Efficiency Protocol · p. 11

D. Implementation Details

The LoRA baselines use rank-16 adaptation on all LLM linear projections, with more trainable parameters and longer convergence training than the geometric head.

D. Implementation Details

  • LoRA uses rank 16 with α = 8 on all LLM linear projections and requires more trainable parameters and training time than the geometric head.The Qwen and LLaVA LoRA variants have 18.46M and 8.80M trainable parameters and take about 15 hours to converge, versus 4.08M parameters and 4 hours for the geometric head.Source: D. Implementation Details · p. 11

E. Efficiency–Performance Trade-off Across Exit Depth

Exit depth trades trajectory inference speed against accuracy, while query-conditioning effects vary across architectures and tasks. The final text token is used across the main experiments for a consistent decoder design.

E. Efficiency–Performance Trade-off Across Exit Depth

F.1. Query Conditioning Strategy

  • Attention pooling improves trajectory prediction on Qwen2-VL-2B but degrades LLaVA-OV-0.5B, whereas the final text token is stronger for affordance prediction on both.The final text token conditions the decoder query in the main experiments to keep the design consistent across architectures and tasks.Source: §F.1 · p. 12, Table 3 · p. 12

F.2. Trajectory Loss Components

Explicit start- and end-point supervision improves trajectory metrics and prevents predictions from collapsing toward a dataset-level mean. The heatmaps show recovery of spatially diverse, input-specific predictions.

F.2. Trajectory Loss Components

  • Adding start/end supervision improves all reported metrics: DFD falls from 0.322 ± 0.02 to 0.251 ± 0.010, HD from 0.26 ± 0.01 to 0.219 ± 0.008, and RMSE from 0.26 ± 0.03 to 0.177 ± 0.007.Source: §F.2 · p. 12
  • Adding start- and end-point supervision restores spatial diversity matching the ground truth and input-dependent localization.The reported improvement spans DFD, HD, and RMSE.Source: Figure 7 · p. 12, §F.2 · p. 12
  • Without endpoint supervision, predictions collapse toward the image center because uniform L1 loss favors the dataset-mean trajectory.The heatmaps contrast this geometry-head behavior with the zero-shot baseline's boundary-collapsed predictions.Source: §F.2 · p. 12, Figure 7 · p. 12

G. Qualitative Results

The qualitative figures show affordance boxes and manipulation trajectories predicted from observations and task instructions using Qwen2-VL-2B and LLaVA-OV-0.5B with Geo-Head. They also include failure cases.

G. Qualitative Results

  • Qwen2-VL-2B + Geo-Head predicts diverse end-effector trajectories in the first three rows; the fourth includes failures from object-localization errors or insufficient spatial understanding.Green curves are ground truth and red curves are predictions.Source: Figure 8 · p. 13
  • LLaVA-OV-0.5B + Geo-Head shows diverse trajectory predictions in the first three rows and failure cases in the fourth.Green curves are ground truth and red curves are predictions.Source: Figure 9 · p. 14
  • Affordance-box visualizations for both backbones show predictions based on the observation and task instruction, alongside failure cases in the fourth row.Green boxes are ground truth and red boxes are predictions.Source: G. Qualitative Results · p. 12, Figure 10 · p. 15, Figure 11 · p. 16

Related papers

Loading 2610.39416v1…