PaperlayerView exact revision on arXiv

Source-linked AI summary

Proprioceptive Sketches as Long-Horizon Intent for Generative Action Policies

Fangyuan Wang, Songhao Huang, Haoxiang Sun, Shipeng Lyu, Chengyang He, Anqing Duan, Peng Zhou, David Navarro-Alarcon

arXiv:2610.02759v1cs.RO

TL;DR

Long-horizon manipulation remains challenging for generative action policies, while dense predicted trajectories can be difficult to represent compactly. PAM jointly generates a timing-free sketch of the remaining robot path and an executable action chunk. It improves over action-only policies across the evaluated tasks, with real-world success rising from 47.5% to 75.0%.

  • Problem

    Generative action policies perform strongly on short-horizon tasks but are less effective on long-horizon manipulation, and dense time-indexed paths are difficult to represent compactly across a full task.

    Source: §1 · p. 2

  • Method

    PAM jointly generates a compact, arc-length-parameterized sketch of the remaining configuration-space path and a short executable action chunk, using directed attention and staggered denoising.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    PAM improves over corresponding action-only policies on Push-T, LIBERO-Long, and four real-world bimanual tasks; on the real-world tasks, it succeeds in 60 of 80 episodes (75.0%) versus Diffusion Policy's 38 (47.5%).

    Source: §4.3 · p. 7, §5 · p. 9

  • Takeaways & Limitations

    The sketch supplies long-horizon geometric context while only the separate action chunk determines the immediate motion and gripper commands.

    Source: §1 · p. 2

    The sketch represents robot motion without modeling object dynamics or contact.

    Source: §5 · p. 10

Abstract

from arXiv · show

Generative robot policies predict short action chunks but lack explicit long-horizon intent. Recent methods expose longer-horizon structure through language plans, subgoal images, or video forecasts, which are costly to generate and still need to be translated into robot motion. Predicting future robot motions avoids this translation, but a dense, time-indexed trajectory requires numerous parameters to cover the full remaining task, and over a short horizon it largely repeats the action chunk and adds little guidance for action generation. We propose Proprioceptive Action Models (PAM), which jointly generate a compact, timing-free sketch of the robot's remaining joint-space path and a dense executable action chunk within a single transformer denoiser. The sketch parameterizes the path by arc length rather than time, capturing geometric intent invariant to execution timing. Block-causal attention and a staggered denoising schedule maintain directed sketch-to-action dependence, ensuring the action tokens condition on a progressively cleaner sketch throughout sampling. In simulation, PAM improves over its action-only counterparts on Push-T and LIBERO-Long; on four real-world bimanual tasks, it raises success from 47.5% to 75.0%. Project page: https://nicehiro.github.io/pam_dp/

1 INTRODUCTION

Generative action policies remain less effective on long-horizon manipulation, while existing intent representations can be costly or difficult to translate into control. The paper introduces a compact, timing-free sketch of the remaining robot path, jointly generated with an executable action chunk.

1 INTRODUCTION

  • Visual forecasts, predictive scene representations, and language plans represent scene evolution or task structure but can be costly or require mappings to continuous control.These limitations motivate a compact long-horizon representation tied to the robot’s own motion.Source: §1 · p. 2
  • A dense, time-indexed trajectory is hard to parameterize over the full task, while a short-horizon prediction largely duplicates the action chunk.The desired representation should cover remaining motion with a fixed number of parameters while adding guidance beyond the action chunk.Source: §1 · p. 2
  • The proprioceptive sketch represents the remaining configuration-space path with arc-length parameterization and a small number of B-spline coefficients.It captures geometric intent without specifying execution speed or duration; only the separate action chunk is executed.Source: §1 · p. 2
  • PAM jointly generates the sketch and an action chunk in one transformer denoiser, with action tokens attending to—but not feeding information back into—the sketch.A staggered schedule denoises the sketch ahead of actions, providing cleaner geometric context during action generation.Source: §1 · p. 2
  • The sketch is invariant to temporal reparameterization and is evaluated in PAM-DP on Push-T and real-world bimanual tasks and in PAM-VLA on LIBERO-Long.The paper applies the joint generator to both a diffusion transformer trained from scratch and a pretrained vision-language-action model.Source: §1 · p. 2

2 RELATED WORK

Prior work represents intent through language, visual or latent predictions, robot-motion trajectories, and jointly generated scene-action outputs. PAM instead predicts a timing-free, full-task robot path jointly with executable actions through directed attention and staggered denoising.

Predictive intermediate representations

  • Language, visual, and latent predictive representations capture task structure or scene evolution, but can be expensive to generate or require a learned decoder to reach continuous control.PAM predicts a configuration-space path from proprioception, requiring no annotation beyond recorded proprioception.Source: §2 · pp. 2–3

Robot-motion representations

  • PAM differs from prior robot-motion approaches by sketching the remaining path to task completion, removing execution timing, and generating the sketch jointly with actions.Earlier approaches include image-plane trajectory sketches, visual traces, future keypoints, end-effector trajectories, and executable spline policies.Source: §2 · p. 3

Joint generation and denoising schedules

  • PAM couples a non-executable motion sketch with an executable action chunk; directed attention lets actions depend on the sketch without feeding action information back into it.Staggered denoising lets the sketch become cleaner ahead of the actions.Source: §2 · p. 3

3 METHOD

PAM represents the robot’s remaining configuration path as compact geometric intent alongside a short executable action chunk. A shared denoiser generates both, with directed attention and sketch-first denoising.

3.1 PROBLEM SETUP

  • PAM models a conditional generative policy over the remaining-path sketch and short action chunk, conditioned on observations, configuration, and an optional task specification.The denoising procedure applies jointly to both token streams using a single transformer denoiser.Source: §3.1 · p. 3

3.2 THE PROPRIOCEPTIVE SKETCH

  • The sketch encodes the remaining configuration path by normalized arc-length phase and cubic B-spline coefficients; stationary intervals add no phase, and zero-length paths map to zero.The phase measures the fraction of path length traversed rather than elapsed time.Source: §3.2 · pp. 3–4
  • Re-timing a fixed continuous path, including pauses, leaves its sketch unchanged for a fixed coefficient budget, though separately recorded paths can differ through sampling and measurement noise.This invariance concerns the underlying continuous path.Source: §3.2 · p. 4

3.3 JOINT SKETCH AND ACTION GENERATION

  • A shared transformer jointly generates sketch and action streams with block-causal attention: sketch tokens cannot read actions, while actions can read the sketch and earlier actions.The sketch ordering follows B-spline basis support along phase, and action ordering follows execution time.Source: §3.3 · p. 4, Figure 3 · p. 5
  • At inference, staggered denoising advances the sketch ahead of actions to provide cleaner sketch inputs; training samples independent noise coordinates for the two streams.Tokens within each stream share its noise coordinate, while the schedule gives the sketch a positive lead over the actions.Source: §3.3 · pp. 4–5, Figure 3 · p. 5

4 EXPERIMENTS

Across Push-T, LIBERO-Long, and four real-world bimanual tasks, the experiments compare PAM with action-only and spline baselines and report higher coverage or success for PAM.

4.1 SETUP

  • Evaluation covers Push-T, ten language-conditioned LIBERO-Long tasks, and four dual-UR3 bimanual tasks; PAM-DP and PAM-VLA are compared with their action-only baselines.The study also compares B-spline Policy, which parameterizes executable actions as spline curves.Source: §4.1 · pp. 5–6

4.2 SIMULATION

  • Push-T last-ten-checkpoint coverage rises from 0.79 to 0.88 with state observations and 0.66 to 0.72 with image observations; LIBERO-Long success rises from FLOWER's 94.9% to 95.2%.The LIBERO-Long standard deviation is 0.4 percentage points across five training seeds.Source: Table 1 · p. 6, §4.2 · pp. 6–7, Table 2 · p. 6

4.3 REAL-WORLD DUAL-ARM MANIPULATION

  • PAM-DP succeeds in 60 of 80 real-world episodes (75.0%), versus 38 (47.5%) for Diffusion Policy and 35 (43.8%) for B-spline Policy.PAM-DP takes slightly longer on successful episodes, consistent with the additional NFE and sketch tokens.Source: Table 3 · p. 7, §4.3 · pp. 7–8
  • In handoff, PAM-DP's sketch extends toward the transfer region near the barrier, while the action chunks from the compared policies remain near the current gripper positions.Figure 5 projects predicted motions into an external camera view; the B-spline Policy's orange squares and curves mark its control points and action spline.Source: Figure 5 · p. 7, §4.3 · p. 8
  • Missed-grasp failures fell from 22 to 10, coordination errors from 8 to 1, and object-loss failures from 12 to 9 versus Diffusion Policy; Figure 6 shows individual grasp retries.The illustrated episodes are examples, not a comparison of overall recovery success rates.Source: Table 4 · p. 8, §4.3 · p. 8, Figure 6 · p. 8

4.4 ANALYSIS

  • With observation and action noise fixed, replacing the handoff sketch route changes predicted left-tool position by 10.5 mm RMS, versus 8.5 mm for action-noise resampling and 1.5 mm for same-route replacement.Back-route sketches induce greater predicted backward motion of the left gripper.Source: Figure 7 · p. 9, §4.4 · p. 9
  • On Push-T, zero-lead PAM-DP coverage was 0.79 versus 0.69 for action-only and 0.75 for auxiliary-only; time-indexed sketches, bidirectional attention, and shared-time training scored 0.74, 0.72, and 0.74.The ablation holds the other configuration fixed and removes staggered denoising to isolate representation and training effects.Source: §4.4 · pp. 8–9, Table 5 · p. 9
  • On Push-T, a sketch lead of η = 1/8 raises coverage from 0.779 to 0.805 with 18 rather than 16 NFE; η = 1 reaches 0.809 with 32 NFE.All tested positive leads yield higher mean coverage than zero lead.Source: Table 6 · p. 9, §4.4 · p. 9

5 CONCLUSION

PAM combines a timing-free B-spline sketch of the remaining configuration path with an executable action chunk, and the sketch improves corresponding action-only policies across the reported tasks. The conclusion also identifies limits in representing object interaction and in attributing gains from staggered denoising.

5 CONCLUSION

  • PAM jointly generates a timing-free B-spline sketch and short action chunk; adding the sketch improves corresponding action-only policies on Push-T, LIBERO-Long, and four real-world bimanual tasks.The sketch represents the remaining configuration-space path; only the action chunk is executed.Source: §5 · pp. 9–10
  • The sketch omits object dynamics and contact, while additional NFE and an unmatched lead sweep leave the denoising-schedule effect unresolved.Future work will explore sketches capturing object interactions and evaluate schedules under matched inference budgets.Source: §5 · p. 10

A.1 PATH PREPROCESSING AND PHASE PARAMETERIZATION

  • For dual-arm path construction, the remaining joint positions are expressed relative to the current configuration, resampled at 64 arc-length phases, and fitted after excluding gripper states.The tolerance removes numerically repeated samples but not encoder jitter, which can contribute small phase increments.Source: §A.1 · p. 13

A.2 B-SPLINE REPRESENTATION AND TARGET FITTING

  • The resampled path is fitted with a clamped cubic B-spline by least squares, with the first and last control points fixed to zero and the terminal displacement.The basis matrix and pseudoinverse are precomputed for each selected pair of control-point and sample counts.Source: §A.2 · pp. 13–14
  • Arc-length parameterization is invariant to speed changes and inserted pauses that preserve traversal order, but not reversal, retracing, or timing changes between arms.Finite sampling, noise, and tolerances can still make separately recorded demonstrations differ.Source: §A.2 · p. 14

A.3 CAPACITY AUDIT AND NORMALIZATION

The capacity audit compares sketch reconstruction error with each task’s per-step motion reference and checks the capacities used in policy experiments. Validation-target clipping causes only the reported small changes in reconstruction RMSE.

A.3 CAPACITY AUDIT AND NORMALIZATION

  • The audit measures scalar RMSE against 64 uniformly spaced phase targets and uses training episodes, with task-specific anchors and adjacent-frame motion increments as references.Push-T and dual-arm tasks use training-sampler anchors; LIBERO uses every frame.Source: §A.3 · p. 14
  • 0.845 normalized reconstruction error for LIBERO-Long at M = 16 (0.0098 rad RMSE versus a 0.0116 rad per-step reference); adopted capacities satisfy Erec(M) < δstep on all six tasks.The audit is post hoc and does not require the smallest budget meeting the criterion.Source: §A.3 · p. 14
  • Validation-target clipping changes reported RMSE only slightly across the four real-world tasks, while Push-T remains at 3.08 pixels at reported precision.These values measure distortion of fitted targets, not model prediction error.Source: §A.3 · p. 14

B MODEL IMPLEMENTATION

This section locates the joint policy’s token processing, training targets, and sampling updates, with experiment-specific settings documented separately.

B MODEL IMPLEMENTATION

  • Token processing, training targets, and sampling updates are specified here; experiment-specific dimensions, optimization settings, and inference budgets are collected in Appendix C.Source: B MODEL IMPLEMENTATION · p. 14

B.1 ARCHITECTURE

The architecture uses separate sketch and action streams processed by a shared transformer, with directed attention and observation conditioning. PAM-VLA adapts the same sketch-prediction idea to FLOWER’s rectified-flow action expert.

B.1 ARCHITECTURE

  • PAM-DP projects sketch coefficients and action steps through separate linear layers into a shared embedding space, then uses a shared transformer decoder and separate output heads.The sketch stream also receives a learned type embedding, and both streams have learned positional embeddings.Source: §B.1 · pp. 14–15
  • Sketch tokens attend causally within their stream and never read actions, while action tokens attend to all sketch tokens and causally to prior actions.Source: §B.1 · p. 15
  • The observation encoder forms cross-attention memory from independently processed image and low-dimensional inputs, without temporal mixing in the conditioning network.Sketch queries read all observation positions; action queries read positions up to their own time index.Source: §B.1 · p. 15
  • PAM-DP separately embeds sketch and action noise levels to modulate and gate decoder-block updates, with modulation projections and output heads initialized to zero.Source: §B.1 · p. 15
  • PAM-VLA adds sketch prediction to FLOWER’s rectified-flow action expert using separate stream encoders and velocity heads, with projected VLM features as cross-attention context.Its pretrained Florence-2 backbone, including the vision tower, is fine-tuned rather than frozen.Source: §B.1 · p. 15

B.2 TRAINING AND SAMPLING

PAM trains its sketch and action streams with independently sampled noise levels and uses staggered inference so the sketch leads action denoising. Push-T uses diffusion, while PAM-VLA uses rectified flow.

Diffusion training

  • Push-T uses a 100-step cosine diffusion schedule with independently sampled sketch and action timesteps, and stream-wise mean-squared noise-prediction losses.Every token in a stream shares its timestep, while noise is sampled independently per token and feature.Source: §B.2 · p. 16

Real-world diffusion settings

  • For real-world experiments, inference uses synchronous 16-step DDIM for checkpoint validation and a sketch lead of η = 1/8 for main results.The independent-time model samples separate stream timesteps; the shared-time control uses one timestep across both streams.Source: §B.2 · p. 16, Table 7 · p. 17

Rectified-flow training

  • PAM-VLA trains with rectified flow, independently sampling one uniform noise coordinate per stream and weighting each stream’s averaged squared velocity error.Its interpolation is α(γ) = 1−γ and σ(γ) = γ, with the velocity target ϵ_i − x0_i.Source: §B.2 · p. 16

Discrete sketch lead

  • Push-T inference offsets sketch denoising by η = 1/8, adding 13 updates to the 100-step schedule for 113 NFE while retaining 100 updates per stream.The sketch advances ahead of action denoising, then remains fixed while action updates finish; the lead is an inference setting, not a change to training corruption.Source: §B.2 · p. 16

Rectified-flow sampling

  • LIBERO PAM-VLA uses K = 8 rectified-flow steps and η = 1/8, yielding nine NFE with Euler updates and outputs clipped to [−1, 1].The stream-wise noise coordinates follow the rectified-flow schedule without discrete lead rounding.Source: §B.2 · p. 16

Execution and ablations

  • After sampling, the policy restores normalizers and the sketch’s zero anchor, executes only the action chunk, and replans; shared-time training and zero-lead sampling are distinct ablations.Push-T predicts 16 action positions from two observation steps and executes eight before replanning.Source: §B.2 · p. 16

C.1 TRAINING AND INFERENCE CONFIGURATIONS

The appendix specifies training schedules, task-specific loss weights, and direct comparison methods for PAM-DP and PAM-VLA.

PAM-DP training settings

  • PAM-DP uses sketch loss weight 0.15 for close lid and 0.2 for the other three real-world tasks, with action loss weight one.The real-world PAM-DP settings are shared across handoff, close lid, fold cloth, and measure.Source: §C.1 · p. 17

PAM-VLA configuration

  • PAM-VLA fully fine-tunes Florence-2-large, including its vision tower, with a staged learning-rate schedule and EMA evaluation weights.The learning rate rises from 2 × 10−6 to 2 × 10−5 over 1,500 updates, then decays to 10−5; EMA maximum decay is 0.9999.Source: §C.1 · p. 17

Inference lead

  • The main PAM-DP results use inference lead 1/8; its 16-step DDIM lead experiment uses 18 NFE.The experiment uses two lead steps in addition to K = 16 sampling steps.Source: §C.1 · p. 17

Baselines

  • Direct comparisons pair PAM-DP with action-only Diffusion Policy and PAM-VLA with FLOWER, while B-spline Policy predicts executable action curves.Real-world methods share the visual encoder, EMA, and optimizer-update count for each task.Source: §C.1 · pp. 17–18

C.2 DATASETS AND EVALUATION PROTOCOLS

The appendix defines demonstration splits and evaluation protocols across simulation and real-world tasks, then describes fixed-observation interventions used to test action dependence on sketches.

C.2 DATASETS AND EVALUATION PROTOCOLS

  • LIBERO uses all 500 demonstrations for training, while validation episodes are excluded from normalizer statistics; Table 9 gives demonstration counts and seed-42 splits.Push-T follows the split of Chi et al. (2023), and LIBERO has no demonstration-level validation split.Source: §C.2 · p. 18, Table 9 · p. 18
  • Push-T reports maximum coverage over checkpoints and the mean of the last ten, with checkpoint aggregates over 50 initial conditions and three training seeds.Component ablations instead use seed 42, 50 initial conditions, 100-step DDPM, and zero lead; their statistics cover the last ten checkpoints, not independent runs.Source: §C.2 · p. 18
  • The lead evaluation uses one seed-42 final-epoch checkpoint and deterministic 16-step DDIM, with paired initial noise across leads and NFE reported per action chunk.It evaluates 50 initial-condition seeds with three sampler-noise seeds; standard deviations are across sampler-noise seeds.Source: §C.2 · p. 18
  • LIBERO-Long success is averaged across ten tasks with 50 trials per task, and the main PAM-VLA table aggregates five training seeds.Baseline results are drawn from original publications, except π0, which uses the result reported by Kim et al. (2025).Source: §C.2 · p. 18
  • Each real-world method runs 20 episodes per task for at most 120 seconds, and checkpoint selection uses offline joint-action RMSE rather than robot evaluation episodes.Completion time averages successful episodes only; episodes unsuccessful by 120 seconds count as failures.Source: §C.2 · p. 18

C.3 SKETCH INTERVENTION PROTOCOL

  • The sketch intervention replaces denoising trajectories with matched-noise donor sketches at one fixed handoff observation, while tool-position displacement measures action dependence, not task success or collision clearance.The intervention compares native and transplanted front-route sketches, back-route donors, and an action-noise control; figure overlays are qualitative and translated paths are for display only.Source: §C.3 · p. 19
Loading 2610.02759v1…