PaperlayerView exact revision on arXiv

Source-linked AI summary

TEMPO: Learning Temporal Context for Dynamic Robot Manipulation

Zhenyang Feng, Jimin Heo, Erik B. Sudderth, Unnat Jain

arXiv:2609.16864v1cs.ROcs.CVcs.LG

TL;DR

Single-observation VLAs struggle with dynamic manipulation because they lack temporal context for both scene motion and task phase. TEMPO augments a pretrained VLA with motion and proprioceptive histories, improving performance across four dynamic tasks while adding minimal overhead and releasing a motion-perception benchmark.

  • Problem

    Single-frame VLAs face motion ambiguity and state aliasing: moving scenes lack encoded dynamics, while visually similar task phases can require different actions.

  • Method

    TEMPO adds TEMPOMOT, a frozen-video-model motion representation, and TEMPOACT, a compact proprioceptive history, without modifying the pretrained backbone.

  • Results

    Across four dynamic tasks, TEMPO improves Bottle Handover success from 44% to 74% and is the only system completing full Wine Pour and Flick Catch tasks.

  • Takeaways & Limitations

    The results support a primarily representational explanation of dynamic-VLA failures, with separate temporal signals addressing motion ambiguity and state aliasing.

  • Takeaways & Limitations

    Quantitative comparisons use a trimmed version of Wine Pour and Flick Catch that removes the release-and-retract phase, making the reported gain a lower bound on full-task advantage.

Abstract

from arXiv · show

Vision-language-action (VLA) models have achieved impressive performance in quasi-static manipulation, but struggle in dynamic manipulation tasks because they operate on a single observation at inference time. We identify two representational failures that underlie this limitation. The first is motion ambiguity, where a single observation does not include scene dynamics and therefore cannot anticipate the future state of moving objects. The second is state aliasing, where visually similar observations from different points in a task require different actions. We argue that these failures persist regardless of model scale and inference latency, showing that the bottleneck is missing temporal context rather than model capacity. Based on this insight, we propose TEMPO, which augments a pretrained VLA with two temporal inputs: a motion summary extracted from a frozen video foundation model to resolve motion ambiguity and a compact proprioceptive history to resolve state aliasing. TEMPO requires no modification to the backbone and adds minimal compute overhead at training or deployment. Across four dynamic manipulation tasks, it improves Bottle Handover success from 44% to 74% and is the only method that solves state aliasing. Probing and ablation studies confirm that each temporal signal independently addresses its corresponding failure. We further release TEMPO-Bench, a benchmark of over 50k annotated frames for evaluating motion-aware robot perception in both regression and multiple-choice formats. Project Website: https://tempo-robot.github.io/

1 Introduction

TEMPO identifies missing temporal context as the central limitation of single-observation VLAs in dynamic manipulation and adds two compact signals to address motion ambiguity and state aliasing. Across four tasks, it improves dynamic performance while preserving the pretrained backbone and introduces a benchmark for motion-aware perception.

  • Single-frame VLAs cannot infer how a moving scene evolves, so they cannot anticipate a target’s future state from the freshest observation alone.This failure is termed motion ambiguity and remains even when inference latency is reduced.
  • Visually similar observations from different task phases can require opposite actions, producing jitter and mistimed commitments through state aliasing.The problem arises because sequential subtasks are not disambiguated by the current frame alone.
  • TEMPO adds TEMPOMOT from a frozen video foundation model and TEMPOACT from recent proprioceptive history without modifying the pretrained backbone.Together, the signals provide scene dynamics and subtask context, adding only 2M parameters, or 0.08% overhead.
  • 44% to 74% Bottle Handover success, while TEMPO is the only system completing full Wine Pour and Flick Catch tasks.The evaluation covers four dynamic tasks and reports the strongest gains where motion perception or state disambiguation limits performance.
  • In Drop Catch, TEMPO tracks the human hand while the baseline oscillates left-to-right despite the hand moving in one direction.The rollout comparison illustrates the motion-aware behavior enabled by TEMPO.
  • TEMPO-Bench releases over 50k annotated human-robot manipulation frames with object-velocity labels and regression and multiple-choice evaluation formats.The benchmark targets motion understanding in vision-language models and VLAs.

2 Related Work

Prior dynamic-robotics systems rely on task-specific physics models and controllers, while asynchronous VLA methods primarily address execution timing. TEMPO instead extends generalist pretrained VLAs with temporal input because existing visual and visuomotor methods largely condition policies on a single frame.

  • Task-specific dynamic-robotics systems achieve strong results in scenarios such as table tennis, tossing, and aerial catching but do not transfer broadly across tasks.These systems use explicit object-physics modeling and specialized controllers in end-to-end pipelines.
  • Asynchronous inference methods target stale observations caused by VLA execution latency, using reduced or corrected staleness and, in one case, anticipated future frames.This line of work addresses timing rather than the missing motion information in a single observation.
  • Visual and visuomotor representation methods learn transferable features, affordances, or heterogeneous-embodiment representations, but condition the policy on a single frame.TEMPO differs by introducing temporal inputs for dynamic manipulation.

3 Motion Ambiguity and State Aliasing

Single-frame VLAs fail in dynamic manipulation because they lack temporal context: they cannot infer object motion or distinguish visually similar task phases requiring opposite actions.

  • Motion Ambiguity: Motion ambiguity prevents a current-frame policy from anticipating object motion, and latency reduction only provides a more recent observation.On motion-only tasks, baseline velocity decoding collapses, with R2 ≤0.03 on Flick Catch and ≤0.09 on Wine Pour.
  • State Aliasing: State aliasing makes visually near-identical moments in multistage tasks require opposite actions, causing hesitation and task failure.The failure persists even without object motion because the current frame does not identify the task phase.
  • Empirical Evidence: On untrimmed Flick Catch and Wine Pour, both asynchronous baselines fail every rollout while TEMPO completes the tasks using TEMPOACT.The retained release-and-retract phase exposes the state-aliasing failure.
  • What the Two Failures Require: Both failures arise because a single frame carries no temporal context, requiring recent visual observations for motion and recent robot actions for task-phase disambiguation.These two signals summarize information already available to the policy during inference.

4 Temporal Encoding for Motion-aware Policy (TEMPO)

TEMPO adds two compact temporal inputs to a pretrained VLA: a visual motion summary from recent frames and a proprioceptive history summary for task-phase context.

  • 4.1 TEMPOMOT: Captures Object and Scene Dynamics: TEMPOMOT cross-attends current-frame features with a rolling cache of preceding-frame features to produce a motion code.A frozen pretrained video foundation model supplies the per-frame features, and cached features are computed once and reused.
  • 4.1 TEMPOMOT: Captures Object and Scene Dynamics: Swapping the motion encoder keeps Bottle Handover success above the VLASH baseline of 38%.Across four video foundation models, success remains in the 52–76% range; the default encoder is selected for streaming speed.
  • Integration: TEMPO attaches both signals to the VLA prefix, while TEMPOACT also conditions the action expert through an AdaRMS residual.The backbone remains largely intact and the added conditioning path is zero-initialized.
  • 4.2 TEMPOACT: Resolves State Aliasing: The two signals provide complementary context: TEMPOMOT captures scene motion, while TEMPOACT captures the robot’s recent behavior to disambiguate task phases.Without proprioceptive history, the policy can average over conflicting valid actions and become indecisive or oscillatory.
  • 4.2 TEMPOACT: Resolves State Aliasing: TEMPOACT partitions recent proprioceptive commands into equal temporal buckets and averages each bucket into a fixed-size history summary.The representation uses K = 10 buckets of L = 30 commands and preserves a coarse motion trend with negligible overhead.
  • Efficiency: TEMPO adds 2M parameters, or 0.08%, and about 1.5 ms of policy forward-time overhead beyond the asynchronous baseline.Median forward time is 35.3 ms versus 33.8 ms, while the motion encoder runs in a separate background thread.

5 Experiments

TEMPO is evaluated across four dynamic manipulation tasks, with success rates, failure modes, backbone transfer, latency, and a compact training-target extension providing complementary evidence.

  • 5.1 Experimental Setup: Table 3 compares inference latency across VLASH, RTC, and TEMPO, while the benchmark covers four comprehensive dynamic tasks.The supplied latency entries are VLASH 33.8 ms, RTC 127.7 ms, and TEMPO 35.3 ms.
  • 5.2 Experimental Results: +36% Bottle Handover success is reported for TEMPO over asynchronous baselines, with further gains on Wine Pour and Flick Catch.Success rates are computed over 50 real-world rollouts per task; Wine Pour uses retained wine mass rather than binary completion.
  • 5.2 Experimental Results: Both asynchronous baselines predominantly fail Bottle Handover through Complete Misses, indicating incorrect prediction of the moving bottle’s location.VLASH and RTC commit 18 and 24 Complete Misses, respectively.
  • 5.2 Experimental Results: TEMPO’s temporal inputs improve Bottle Handover across asynchronous and non-asynchronous backbones, including π0.5 from 26% to 56% and RTC from 44% to 80%.The same TEMPOMOT and TEMPOACT inputs are attached to additional pretrained VLAs and retrained on the same demos.
  • 5.2 Experimental Results: TEMPO+UVT achieves the highest overall success rate and outperforms baselines on 3 of 4 tasks by combining temporal inputs with a compact motion-primitive target.TEMPO addresses the input side, while UVT changes the output-side training target.

6 Analysis and Ablations

Ablations and probing test whether TEMPO’s two temporal signals are independently useful and whether they leave motion information in the policy representation.

  • Analysis approach: The analysis evaluates signal reliance through single-signal ablations and hidden-state probing at task stages where the corresponding ambiguities arise.The probe benchmark uses per-frame object positions and derived pixel-space velocity over more than 50k annotated frames.
  • Input ablations: Removing either TEMPOMOT or TEMPOACT substantially reduces Bottle Handover success, while combining both signals reaches 74%.Each single-signal variant still outperforms the 38% VLASH baseline, showing that both inputs contribute non-redundant information.
  • Hidden-state probing: TEMPO’s hidden state decodes object velocity better than asynchronous baselines on every probed task, especially Flick Catch at R2 = 0.57 versus ≤0.03.On Wine Pour, TEMPO reaches R2 = 0.44 versus ≤0.09 for the baselines.

7 Conclusion

The conclusion frames dynamic VLA performance as a representational problem addressed by two plug-and-play temporal inputs, supported by separable failure analyses and benchmark release.

  • 7 Conclusion: TEMPO’s conclusion attributes the dynamic-manipulation gap primarily to motion ambiguity and state aliasing rather than timing.The cited conclusion states that each failure has a corresponding representation and that the signals address them separately.
  • 7 Conclusion: The method implements the proposed fix as two plug-and-play inputs with near-zero overhead.The conclusion links the inputs to gains on untrimmed Wine Pour and Flick Catch, where asynchronous baselines score zero.
  • Appendix scope: The appendix supplements the paper with qualitative rollouts, failure-mode analyses, experimental setup details, input ablations, and motion-encoder selection.These materials are organized across Sections A–E and include additional task-level and encoder analyses.

A Additional Failure Mode Analysis

Additional experiments distinguish state aliasing from motion ambiguity by comparing trimmed tasks with the original untrimmed episodes.

  • Task trimming: The release-and-retract segment is trimmed for the non-trivial baseline comparison, preserving target tracking and timing while removing the state-aliased segment.Both asynchronous baselines otherwise collapse to 0% on the complete tasks.
  • Untrimmed-task evaluation: On the original untrimmed tasks, RTC and VLASH fail every rollout, whereas TEMPO reaches 68% on Flick Catch and 97.6% on Wine Pour.The untrimmed evaluation directly tests whether TEMPO handles the release-and-retract phase rather than benefiting only from trimming.
  • State-aliasing interpretation: The untrimmed gains are attributed to TEMPOACT, whose proprioceptive history distinguishes task phases that appear similar in the current frame.This lets the policy commit to the appropriate reach-out or retract action instead of oscillating between them.

B Experimental Setup Details

Experiments use a bimanual two-arm platform with three 224p cameras operating at 30 Hz. Evaluation includes controlled dynamic catching setups and comparable training conditions across policies.

  • Hardware: The platform uses two 7-DoF robot arms with parallel-jaw grippers mounted 24 inches apart on a rigid table.Both arms are I2RT YAM Ultra systems.
  • Sensing: Three cameras—one third-person and two wrist-mounted—stream 224p video at 30 Hz, matching the 30 Hz policy rate.
  • Drop Catch: Drop Catch releases a tennis-ball-sized velcro-compatible ball from 13–18 inches above the workspace for capture by a held velcro plate.
  • Flick Catch: Flick Catch places 5–6 identical balls on a shelf near the table edge opposite the robot arms.
  • Controls: Human ball releases use closed eyes and headphones to prevent anticipation of the robot’s position from biasing release points.
  • Training: Each policy trains for approximately 2 hours on eight NVIDIA RTX PRO 6000 Blackwell GPUs with effective batch size 128.Per-iteration times are 0.49 s for VLASH, 0.46 s for RTC, and 0.50 s for TEMPO.

C Per-Task Failure Mode Breakdown

Failure analyses show that baseline errors concentrate in task-specific tracking or timing failures, while TEMPO reduces the dominant failure modes across Flick Catch and Drop Catch.

  • Evaluation: Flick Catch decomposes each rollout into success or failure at plate grabbing, hand tracking, or catching the flicked ball.The analysis covers 50 rollouts per method on the trimmed task.
  • Flick Catch: TEMPO reduces Flick Catch hand-tracking failures from 37 to 9 and ball-catch failures from 19 to 4, dispersing residual errors across all three stages.VLASH primarily fails during hand tracking, whereas RTC primarily loses the flicked ball.
  • Drop Catch: TEMPO cuts Drop Catch premature-movement failures nearly 4×, from 23 to 6 trials, while overshoots change only slightly from 6 to 7.RTC has the highest score on this timing-dominated task, with only a few overshoots and lags.

D Input Ablation

Ablations and probing indicate that TEMPO’s motion and proprioceptive signals contribute at different points in an episode and encode information unavailable from a single position.

  • Modality Ablation: Freezing either temporal modality substantially worsens action prediction, with TEMPOMOT most important at 2.7 s and TEMPOACT at 5.3 s.The ablation evaluates per-frame action MSE on Flick Catch by freezing each modality to its first-frame value.
  • TEMPOMOT: At 2.7 s, freezing TEMPOMOT raises MSE from 0.019 to 0.059, a 3.1× increase, while freezing TEMPOACT costs 0.033.This interval corresponds to the beginning of the task, when large motions occur.
  • TEMPOACT: At 5.3 s, freezing TEMPOACT causes the largest degradation during the finger sweep across balls before selection and flicking.
  • Motion Probe: Every learned motion encoder decodes object velocity far above the raw-position baseline, whose average R2 is 0.08.SAM 2 and VidTwin score highest; SAM 2.1-Tiny is selected for near-top motion correlation and streaming speed.
Loading 2609.16864v1…