PaperlayerView exact revision on arXiv

Source-linked AI summary

PEEK: Guiding and Minimal Image Representations for Zero-Shot Generalization of Robot Manipulation Policies

Jesse Zhang, Marius Memmel, Kevin Kim, Dieter Fox, Jesse Thomason, Fabio Ramos, Erdem Bıyık, Abhishek Gupta, Anqi Li

arXiv:2509.18282v1cs.ROcs.AIcs.LG

TL;DR

Robot manipulation policies often learn attention, action choice, and execution together, and can struggle with novel objects, clutter, and semantic variation. PEEK fine-tunes VLMs to predict paths and task-relevant masks that annotate observations for low-level policies. In real-world evaluations, it improves zero-shot performance across 3D, large VLA, and small policy settings.

  • Problem

    Imitation-learned manipulation policies can degrade on novel objects, clutter, or semantic variations while learning where to attend, what to do, and how to act together.

    Source: §I · p. 1

  • Method

    PEEK fine-tunes VLMs to predict end-effector paths and task-relevant masks, which are overlaid on observations so policies can focus on low-level execution.

    Source: §I · p. 1

  • Results

    41.4× higher real-world success for simulation-trained 3DDA, while π0 and small transformer policies see 2–3.5× success-rate improvements across 535 evaluations and 17 task variations.

    Source: §I · p. 2

  • Takeaways & Limitations

    Real-world evaluations show improved zero-shot generalization across policies guided by PEEK’s minimal, annotated observations.

    Source: §I · p. 2, §VI · p. 6

    PEEK inherits underlying VLM biases and may produce incorrect annotations or fail in out-of-distribution scenarios; its current representation is limited to 2D paths and masks.

    Source: §VI · p. 6

Abstract

from arXiv · show

Robotic manipulation policies often fail to generalize because they must simultaneously learn where to attend, what actions to take, and how to execute them. We argue that high-level reasoning about where and what can be offloaded to vision-language models (VLMs), leaving policies to specialize in how to act. We present PEEK (Policy-agnostic Extraction of Essential Keypoints), which fine-tunes VLMs to predict a unified point-based intermediate representation: 1. end-effector paths specifying what actions to take, and 2. task-relevant masks indicating where to focus. These annotations are directly overlaid onto robot observations, making the representation policy-agnostic and transferable across architectures. To enable scalable training, we introduce an automatic annotation pipeline, generating labeled data across 20+ robot datasets spanning 9 embodiments. In real-world evaluations, PEEK consistently boosts zero-shot generalization, including a 41.4x real-world improvement for a 3D policy trained only in simulation, and 2-3.5x gains for both large VLAs and small manipulation policies. By letting VLMs absorb semantic and visual complexity, PEEK equips manipulation policies with the minimal cues they need--where, what, and how. Website at https://peek-robot.github.io/.

I. INTRODUCTION

PEEK separates high-level decisions about where to attend and what to do from low-level policy execution, addressing the entangled sources of poor generalization in imitation-learned manipulation. It uses a VLM-predicted path and task-relevant masks as a policy-agnostic representation, and reports improved real-world zero-shot performance across policy types.

I. INTRODUCTION

  • Imitation-learned policies can degrade on novel objects, clutter, and semantic variation because attention, object choice, and motion are learned together.For example, grasping a distractor can reflect misplaced attention, an incorrect object choice, and a wrong motion.Source: §I · p. 1
  • PEEK fine-tunes VLMs to predict end-effector paths for what to do and masks for where to focus, drawing both onto observations for arbitrary image-input policies.The policy then specializes in how to execute low-level actions.Source: §I · pp. 1–2
  • 41.4× higher real-world success for a simulation-trained 3DDA policy, while π0 and small transformer policies improve 2–3.5× across 535 evaluations and 17 task variations.The reported gains are for policies guided by PEEK.Source: §I · p. 2

II. RELATED WORKS

  • Unlike masking-only approaches, PEEK also predicts action paths for semantic variation; unlike architecture-specific implicit masking, its explicit masks can be used with policies such as π0 [3].The paper contrasts PEEK with ARRO [13], OTTER [16], and architecture-tied path-prediction approaches, –.Source: §II · p. 2
  • Prior object-centric slot methods generally do not work zero-shot and have not been applied to real-world manipulation; PEEK uses a pretrained VLM to predict relevant points on new objects and tasks.Language relabeling and learned behavioral priors are described as complementary to PEEK’s image-based representation –.Source: §II · p. 2

III. PEEK: GUIDING AND MINIMAL IMAGE REPRESENTATIONS

PEEK offloads high-level task reasoning to a VLM, which produces a guiding and minimal image representation for a low-level policy. The representation specifies what to do with a 2D gripper path and where to focus with task-relevant masks.

III. PEEK: GUIDING AND MINIMAL IMAGE REPRESENTATIONS

  • PEEK shifts high-level task reasoning to a VLM that provides what-to-do guidance and where-to-focus cues, leaving the low-level policy to execute actions.The representation is instantiated as 2D gripper paths and task-relevant masks.Source: §III · p. 2

A. Conceptual Insight

  • The representation is intended to support zero-shot generalization to new scenes, visual clutter, new objects, and unseen language instructions beyond the training dataset.The policy is trained on demonstrations, while the VLM is fine-tuned to produce the guiding minimal representation.Source: A. Conceptual Insight · p. 2
  • The VLM maps task instructions and observations to normalized 2D end-effector path points and an unordered set of task-relevant masking points.Path points trace end-effector positions from the current timestep to the trajectory endpoint; masking points mark relevant objects and regions.Source: A. Conceptual Insight · p. 2
  • Because pretrained VLMs can struggle to predict robot paths and masking points without fine-tuning, PEEK uses scalable labeling across 21 robotics datasets and 9 embodiments.The labeling pipeline produces over 2M VQA pairs spanning 148k trajectories.Source: A. Conceptual Insight · p. 3

B. VLM Data Preparation

PEEK prepares VLM training data by combining existing point-prediction and VQA resources with automatically labeled robot datasets. Its tracking pipeline derives task-relevant points and gripper paths from robot videos.

B. VLM Data Preparation

  • The VLM training mix combines RoboPoint point-prediction and VQA examples with 20 Open X-Embodiment datasets and LIBERO-90 simulation data.RoboPoint contributes 770k point-prediction tasks and 665k VQA examples; the robotics data broadens scene and embodiment coverage.Source: B. VLM Data Preparation · p. 3
  • A visual point tracker follows a grid through each video, and points that move significantly are selected as task-relevant points for the representation.These points correspond to the robot arm or manipulated objects and are tracked across trajectory timesteps.Source: B. VLM Data Preparation · p. 3
  • To encode future task-relevant movement, the pipeline includes tracked points at the final timestep and tracks the gripper to obtain end-effector points.Trajectories are split into subtrajectories at object manipulations, from which 2D gripper paths are constructed.Source: B. VLM Data Preparation · p. 3
  • The VLM’s text target combines shortened path coordinates with task-relevant masking-point coordinates in a trajectory-and-mask format.The passage gives the target form as “TRAJECTORY: [...] MASK: [...].”Source: B. VLM Data Preparation · p. 3

C. VLM and Policy Training/Inference with PEEK

PEEK fine-tunes a VLM to generate paths and masks, then overlays those annotations on images supplied to a low-level policy. At deployment, the VLM is queried periodically and the policy acts between queries.

C. VLM and Policy Training/Inference with PEEK

  • VILA-1.5-3B is fine-tuned for one epoch on 3.5M samples, using supervised log-likelihood prediction of the text answers.Training takes about 20 hours on eight NVIDIA A100 GPUs.Source: C. VLM and Policy Training/Inference… · p. 3
  • The policy receives an augmented image with the predicted path drawn as connected, time-shaded line segments and a mask that reveals image regions around task-relevant points.Each revealed square is centered on a predicted masking point and has edge length equal to 8% of the image size.Source: C. VLM and Policy Training/Inference… · p. 3
  • The VLM is queried every H steps, and the same path and mask are applied to incoming observations until the next query.The hybrid query schedule is intended to balance inference speed and responsiveness while matching training and deployment.Source: C. VLM and Policy Training/Inference… · pp. 3–4, Figure 2 · p. 4
  • Policy training uses the annotated trajectories with the policy’s original objective, while the policy runs at its own inference speed between VLM queries.A VLM query takes about 4–6 seconds on an RTX 3090 without explicit speed optimization.Source: C. VLM and Policy Training/Inference… · p. 4

IV. EXPERIMENTAL SETUP

The experiments test PEEK across real-world robot embodiments, policy classes, and visual and semantic task variations. Evaluation uses simulation-generated or publicly available data and custom setups.

Evaluation scope

  • PEEK is evaluated on two real-world embodiments with 2D policies π0 and ACT and the 3D policy 3DDA, including fine-tuning and training from scratch.The setups test zero-shot generalization from publicly available and simulation-generated datasets while varying task semantics and visual clutter.Source: §IV · p. 4

Franka Sim-to-Real

  • The Franka setup uses 2.5k motion-planned cube-stacking trajectories generated in MuJoCo, with real-world RGB and processed stereo depth.The robot is a Franka Emika Panda with depth processed from Zed 2 stereo-camera RGB images.Source: §IV · p. 4
  • Franka evaluation proceeds from four fixed BASIC cube configurations to CLUTTER and three SEMANTIC tasks, with 220 evaluations across four methods and 11 variations.Each policy receives five trials per task.Source: §IV · p. 4

WidowX BRIDGE

  • The study re-labels single-camera-angle BRIDGE-v2 data with PEEK according to Section III-C and zero-shot evaluates it on a setup with a different table, objects, and background wall.Source: §IV · p. 4
  • WidowX evaluation tests BASIC generalization, added visual CLUTTER, and SEMANTIC tasks requiring visual-language reasoning, with five trials per task and randomized object locations.The 3DDA baselines include HAMSTER paths and ARRO masks, while ACT and π0 are used to test PEEK on 2D policies.Source: §IV · p. 4

Baselines

  • The evaluated baselines include a 90M-parameter ACT transformer conditioned on language embeddings, a 3.5B-parameter π0 VLA LoRA-fine-tuned on BRIDGE, and a 400M-parameter OTTER transformer.OTTER implicitly masks observations by discarding image patches with low CLIP-feature alignment to the task instruction.Source: §IV · p. 4

V. EXPERIMENTAL RESULTS

The results section evaluates PEEK's generalization, its guidance for where to attend and what to do, and the contributions of its components. Figures show evaluation scenes and real-world results for Franka Sim-to-Real and WidowX BRIDGE.

Evaluation questions

  • The evaluation is organized around how much PEEK improves generalization, whether it answers where and what, and how much each component contributes.The questions are addressed in that order.Source: §V · p. 4

Evaluation figures

  • Figure 3 presents Franka zero-shot evaluation environments alongside PEEK path-drawn and masked images, identifying generated simulation data used for policy training.Source: Figure 3 · p. 5
  • Figure 4 presents WidowX evaluation environments alongside PEEK path-drawn and masked images.Source: Figure 4 · p. 5
  • Figure 5 reports task completion rates, including partial credit for correct grasping or reaching, and task success rates across BASIC, CLUTTER, and SEMANTIC variants.It separates Franka Sim-to-Real results at the top from WidowX BRIDGE results at the bottom; values are averaged over trials and tasks within each variant.Source: Figure 5 · p. 5

A. Q1: Real-World Zero-Shot Generalization Experiments

In real-world evaluations, PEEK improves zero-shot success across Franka and WidowX policies. Paths and masks provide complementary guidance: masks reduce visual distractions, while paths guide actions and support semantic generalization.

A. Q1: Real-World Zero-Shot Generalization Experiments

  • 41.4× higher overall success than vanilla 3DDA and 2× over 3DDA+HAMSTER were achieved by 3DDA+PEEK; ACT+PEEK and π0+PEEK reached 3.4× and 2.5× their base models, respectively.π0+PEEK overall achieved a 4.5× better success rate; HAMSTER had 0% partial success on CLUTTER despite 40% partial success on SEMANTIC.Source: A. Q1: Real-World Zero-Shot… · pp. 5–6
  • PEEK masks usually hide task-irrelevant objects, while its paths guide object pickup; ARRO often includes irrelevant objects, which can confuse 3DDA.PEEK's VLM generalizes to new objects, and its paths can guide the policy even when irrelevant object parts remain visible.Source: A. Q1: Real-World Zero-Shot… · p. 5

B. Q2: Does PEEK answer the where and what?

C. Q3: How does each component contribute?

The component ablations find the strongest policy gains when paths and masks are combined, while VLM design tests favor joint prediction with a smaller model. The conclusion notes that PEEK remains bounded by VLM errors and a 2D representation.

C. Q3: How does each component contribute?

  • On simulated cube stacking, paths alone improve performance by +19.3% and masks alone by +32.1%; masks outperform paths because they remove the distractor cube while paths leave ambiguity.The language-conditioned base policy can stack cubes but may ignore instruction order.Source: C. Q3: How does each component… · p. 6
  • Combining paths and masks gives +7.9% over paths, +20.8% over masks, and +40.1% over the base policy on the cube-stacking task.Table I is the paths-and-masks success-rate ablation.Source: C. Q3: How does each component… · p. 6, Table I · p. 6
  • Reducing the VLM from 13B to 3B yields the same DTW 0.12 and IoU 0.68, while joint path-and-mask prediction gives a +19.3% relative gain over mask-only IoU 0.57 without degrading path DTW 0.12.Adding RoboPoint slightly improves the metrics and preserves semantic reasoning ability.Source: C. Q3: How does each component… · p. 6

VI. CONCLUSION AND LIMITATIONS

  • PEEK inherits underlying VLM biases and limitations, which may cause out-of-distribution failures or incorrect annotations.The current representation is limited to 2D point paths and masks; the authors identify richer 3D or multimodal cues and broader video data as future directions.Source: §VI · p. 6

APPENDIX I

The annotation pipeline tracks task-relevant points and the gripper, then splits long trajectories into shorter manipulation segments. These steps produce gripper paths and task-relevant masks for each subtrajectory.

Tracking Points

  • CoTracker3 tracks points from a grid initialized midway through each trajectory; points moving less than 5% of image size are discarded as unlikely to be task-relevant.The remaining points indicate objects that move during the trajectory and the robot gripper.Source: A. Data Annotation Pipeline Details · p. 8

Tracking the Gripper

  • The gripper detector runs on images retaining task-relevant points, reducing noisy detections; missed detections are filled from adjacent frames before averaging the boxes into gripper points.The detector is Detectron2 fine-tuned for end-effector detection.Source: A. Data Annotation Pipeline Details · p. 8

Segmenting into Subtrajectories

  • Long-horizon masks can reveal final placement locations too early, so trajectories are split into subtrajectories using changes in how many task-relevant points stop moving.The pipeline counts points stationary over the next five frames, or three for BRIDGE, and uses two-cluster K-means to distinguish reaching from manipulation.Source: A. Data Annotation Pipeline Details · p. 8
  • Continuous manipulation sections are identified from the stopped-point clusters, and their midpoints become split points for shorter-horizon VLM prediction targets.The resulting segments end when one manipulation finishes and begin before the robot moves to the next object manipulation.Source: A. Data Annotation Pipeline Details · p. 8

B. VLM Dataset Details

The appendix describes dataset preparation and implementation choices, then reports VLM ablations comparing model size, prediction targets, and training data. The reported comparisons favor joint path-and-mask prediction and adding the full OXE and RoboPoint data over BRIDGE alone.

B. VLM Dataset Details

  • OXE subtrajectories are resampled five times with start and end points varied within the first and last 20% of each trajectory; several datasets are excluded, and LIBERO-90 includes 3,958 re-rendered demonstrations across 50 tasks.The first 20% of each full trajectory is also discarded; LIBERO-90 is re-rendered at 256×256 resolution.Source: B. VLM Dataset Details · p. 8
  • The VLM’s autoregressive inference is shortened by simplifying predicted paths and masks with the Ramer–Douglas–Peucker algorithm, using tolerances ϵ = 0.05 and ϵ = 0.1, respectively.Policy training and rollout query intervals differ by dataset, while policies predict action chunks of length five or eight.Source: B. VLM Dataset Details · p. 8, C. VLM and Policy Implementation Details · p. 8

C. VLM and Policy Implementation Details

  • For BRIDGE and Franka Sim-to-Real, the VLM labels policy-training examples at intervals of 30 and 32 timesteps and is queried during rollouts every 25 and 32 timesteps, respectively.Policies predict action chunks of length five for BRIDGE and eight for Sim-to-Real.Source: C. VLM and Policy Implementation Details · p. 8

D. VLM Ablation Results

  • The 3B and 13B models differ minimally, joint path-and-mask prediction improves over either target alone, and adding full OXE plus RoboPoint data helps over BRIDGE alone.Table II evaluates paths and masks on 1,000 hold-out BRIDGE samples.Source: D. VLM Ablation Results · pp. 8–9, Table II · p. 9

E. Full Results Tables

The section points readers to full Franka Sim-to-Real and WidowX BRIDGE results tables. The tables report task success or completion rates averaged over five trials.

E. Full Results Tables

Related papers

Loading 2509.18282v1…