PaperlayerView exact revision on arXiv

Source-linked AI summary

TempoBridge: Language-Guided Tempo Control for Vision-Language-Action Policies

Yeonseo Lee, Hyosup Shin, Guebin Hwang, Sungho Jo

arXiv:2610.09451v1cs.RO

TL;DR

VLA models understand what task to perform but offer limited control over how it is executed. TempoBridge reads tempo cues from frozen VLA representations, aligns them with task phase, and modulates nominal actions. Across LIBERO tasks, it improves tempo compliance while retaining task competence and also demonstrates language-conditioned tempo modulation on a physical robot.

  • Problem

    VLA models provide limited control over execution manner, and connecting semantic tempo cues to execution-time control without tempo-specific demonstrations or policy fine-tuning remains a challenge.

    Source: §I · p. 1

  • Method

    TempoBridge reads tempo from frozen VLA language representations, routes the requested tempo by task phase, and modulates nominal actions without tempo-conditioned demonstrations or base-policy fine-tuning.

    Source: §I · pp. 1–2

  • Results

    Across 40 LIBERO tasks, Tempo Success Rate rises from 52.6% to 89.7%; performance remains near-baseline without cues and improves for unseen tempo expressions.

    Source: §I · p. 2, C. Task Competence without Tempo Cues · p. 6, D. Generalization to Unseen Tempo… · p. 6

  • Takeaways & Limitations

    The results show that language-specified tempo can modulate execution speed in LIBERO and real-world manipulation while largely preserving task competence.

    Source: §V · p. 8

Abstract

from arXiv · show

Vision-Language-Action (VLA) models are effective at understanding what task to perform, but provide limited control over how it should be executed, such as moving quickly or slowly. We introduce TempoBridge, a lightweight framework that uses frozen VLA representations to modulate actions according to tempo cues in the instruction at each task phase, without additional tempo-conditioned robot demonstrations or tempo-specific base-policy fine-tuning. TempoBridge extracts tempo cues from contextual VLM representations, aligns them with task progress through a causal phase router, and modulates nominal motion commands during execution. Across LIBERO tasks, TempoBridge improves Tempo Success Rate from 52.6% to 89.7% under canonical tempo instructions while retaining high task success. It also preserves near-baseline performance when no tempo cue is present and generalizes to unseen tempo expressions without additional training. Experiments on a physical robot further demonstrate language-conditioned tempo modulation in real-world manipulation.

I. INTRODUCTION

VLA models understand task goals but offer limited control over execution manner, while demonstrations and fine-tuning for tempo are costly. TempoBridge reads tempo cues from frozen VLA representations and routes them by task phase to modulate motion; it improves tempo compliance while largely preserving task competence.

I. INTRODUCTION

  • Collecting demonstrations at different speeds is costly and difficult to scale, while using augmented trajectories still requires computationally expensive fine-tuning.Source: §I · p. 1
  • TempoBridge reads semantic tempo from frozen VLA language representations and connects it to phase-aware motion modulation.The base policy continues to generate nominal task actions, which TempoBridge modulates during closed-loop execution.Source: §I · pp. 1–2
  • The framework keeps the base policy fixed and requires neither tempo-conditioned robot demonstrations nor tempo-specific fine-tuning.Its readout uses text examples, and its phase router is trained on existing demonstrations annotated with phase-transition intervals, without tempo labels; unseen tempo expressions can be read without retraining.Source: §I · p. 2
  • Tempo Success Rate rises from 52.6% to 89.7% under canonical quickly/slowly instructions across 40 LIBERO tasks.TempoBridge also preserves near-baseline performance without tempo cues and improves compliance for unseen expressions.Source: §I · p. 2
  • The work evaluates tempo compliance, task competence, and transfer across 40 LIBERO tasks, with additional physical-robot experiments.Source: §I · p. 2

II. RELATED WORK

Prior VLA policies generally leave execution tempo implicit in demonstrations, while existing approaches modify training data or intervene on internal activations. TempoBridge instead reads tempo from language through frozen representations and connects it to phase-aware motion modulation without tempo-specific demonstrations or fine-tuning.

A. Vision-Language-Action Models

  • In VLA policies, language specifies task goals, but execution characteristics such as speed or caution remain implicit in the demonstration distribution.Source: A. Vision-Language-Action Models · p. 2

B. Controlling Execution in Pretrained VLAs

  • Data- and weight-modification approaches learn linguistic-modifier-to-execution mappings from robot trajectories, and TempoVLA still requires policy retraining and an external numeric speed input.Source: B. Controlling Execution in Pretrained… · p. 2
  • Frozen-activation interventions can require manual feature inspection or labeled robot data, while their target is externally specified rather than read from language.TempoBridge reads tempo from the instruction and connects it to phase-aware motion modulation without tempo-conditioned demonstrations or tempo-specific VLA fine-tuning.Source: B. Controlling Execution in Pretrained… · p. 2

III. METHOD

TempoBridge uses a frozen π0.5 policy for nominal actions and adds a tempo readout, causal phase router, and action modulation. It caches tempo sequences from instructions and uses task progress to select which tempo to apply.

A. Overview and Problem Formulation

  • TempoBridge uses a frozen π0.5 policy as its base and adds a prototype-based tempo readout, causal phase router, and tempo-conditioned action modulation.Source: A. Overview and Problem Formulation · pp. 2–3
  • At replanning, the causal phase router uses visual features reused from the policy forward pass and robot state to track progress and select the active event.Source: Figure 2 · p. 3
  • The readout processes an instruction once and caches a tempo sequence, while the router selects its active entry as task progress advances.A single-entry sequence applies one tempo throughout; a two-entry sequence assumes one forward transition between successive events.Source: A. Overview and Problem Formulation · p. 3
  • The selected tempo modulates the nominal action chunk, while readout and router use different representations from the same frozen VLM backbone.The readout runs once on language; the router reuses visual features at each replan, and neither component requires tempo-conditioned demonstrations or base-policy fine-tuning.Source: A. Overview and Problem Formulation · p. 3

B. Prototype-Based Tempo Readout

TempoBridge reads contextual language features from a frozen VLA to detect tempo cues. Its readout uses simple prototypes and fixed thresholds calibrated against tempo-free instructions.

B. Prototype-Based Tempo Readout

  • A single language-only pass through the frozen PaliGemma backbone produces contextual token representations for tempo readout.Valid tokens attend bidirectionally to the instruction, incorporating surrounding linguistic context.Source: 1) Contextual Features and Tempo… · p. 3
  • The readout uses residual-stream features after the twelfth transformer block, with padding and boundary tokens excluded.Training uses 30 LIBERO-90 instructions to construct prototypes and thresholds, with 10 instructions from distinct tasks reserved for validation.Source: 1) Contextual Features and Tempo… · p. 3
  • Unit-norm FAST and SLOW prototypes are built by normalizing and averaging canonical cue-token features, without gradient-based readout training.For each instruction, quickly and slowly are prepended to form cue examples, while the unmodified instruction is tempo-free.Source: 1) Contextual Features and Tempo… · p. 3
  • Each tempo class gets a fixed detection threshold midway between mean cue-feature similarity and mean similarity across tempo-free tokens.The tempo-free mean is computed over individual tokens rather than sentence-level averages.Source: 1) Contextual Features and Tempo… · p. 4

2) Token-Level Tempo Classification:

TempoBridge classifies instruction tokens as FAST, SLOW, or OTHER, then corrects conflicting subtoken labels at the word level. It decodes the order of detected tempo cues; the phase router selects which cue is active during execution.

2) Token-Level Tempo Classification:

  • Token classification combines absolute prototype similarity with competition between FAST and SLOW; OTHER marks a token not selected as a cue, unlike NORMAL, the default execution tempo.Source: 2) Token-Level Tempo Classification · p. 4
  • When subtokens within a word receive conflicting FAST and SLOW labels, their summed threshold excess determines the word label, with ties assigned OTHER.Words without a conflict retain their preliminary labels.Source: 2) Token-Level Tempo Classification · p. 4

3) Tempo Sequence Extraction:

  • After word-level correction, the readout keeps the region with the largest mean threshold excess for each detected tempo class and orders selected regions by midpoint.A single detected class yields a one-tempo sequence; no detected class yields NORMAL.Source: 3) Tempo Sequence Extraction · p. 4
  • For multi-tempo instructions, the decoded sequence preserves cue order rather than assigning modifiers to verbs, and the causal phase router chooses the active entry.Source: 3) Tempo Sequence Extraction · p. 4

C. Causal Phase Routing

For two-entry tempo sequences, a causal phase router aligns the active tempo with task progress online, without oracle phase information at deployment. It estimates a one-way transition from Event 1 to Event 2 using visual features and robot-state history.

C. Causal Phase Routing

  • For two-entry sequences, the recurrent router aligns tempo to task events online without oracle phase information and permits only a forward transition from Event 1 to Event 2.Single-entry sequences bypass the router.Source: C. Causal Phase Routing · p. 4

1) Reused Visual Features and State History:

  • The router combines reused 2048-dimensional visual features, projected to 32-dimensional PCA features, with 8-dimensional robot state and their step-to-step differences.The visual features come from the policy's existing image-token outputs, avoiding an additional visual-backbone pass; differences start at zero and projection statistics are fixed after fitting on router training data.Source: 1) Reused Visual Features and State… · p. 4

2) Causal Transition Estimation:

  • A GRU predicts transition hazards that accumulate into a monotonic transition probability; execution switches to Event 2 when that probability reaches 0.5.The one-way transition leaves Event 2 active for the remainder of the rollout.Source: 2) Causal Transition Estimation · p. 4

3) Interval-Supervised Router Training:

The router is trained on existing demonstrations with reviewed transition intervals, while the VLA feature extractor stays frozen. During execution, tempo selects a fixed gain that scales nominal translation commands, leaving rotation and gripper commands unchanged.

3) Interval-Supervised Router Training:

  • Router training uses 40 demonstrations from 20 LIBERO-90 tasks and validation uses 20 demonstrations from 10 additional tasks, with annotated transition intervals.Intervals are automatically proposed and manually reviewed; the training and validation tasks span object transfer, drawer manipulation, stove operation, and bowl stacking.Source: 3) Interval-Supervised Router Training · p. 4
  • Transition intervals accommodate ambiguous semantic boundaries, and the training objective uses the cumulative transition probability within each annotated interval.For interval (L_n, R_n], the probability of transition within it is F_Rn − F_Ln.Source: 3) Interval-Supervised Router Training · p. 4
  • Only router parameters are optimized on the 40 training demonstrations; the VLA feature extractor remains frozen and the router stays fixed at deployment.Source: 3) Interval-Supervised Router Training · p. 4

D. Tempo-Conditioned Action Modulation

  • Fixed heuristic gains are 1.3 for FAST, 1.0 for NORMAL, and 0.7 for SLOW, symmetric around 1.0 and without task-specific tuning.For two-entry sequences, the gain changes when the phase router switches events; a single-entry sequence uses the same gain throughout.Source: D. Tempo-Conditioned Action Modulation · p. 5
  • TempoBridge scales nominal Cartesian translation commands uniformly across each predicted action chunk, while retaining predicted rotation and gripper commands.The robot executes the first M ≤ H actions, observes again, and replans; realized motion arises through closed-loop interaction rather than a directly prescribed TCP speed.Source: D. Tempo-Conditioned Action Modulation · p. 5

IV. EXPERIMENTS

TempoBridge is evaluated for tempo realization, task competence, behavior without tempo cues, transfer to unseen expressions, and component contributions. The experiments use matched comparisons with the same LIBERO-trained policy and define separate measures for task success, tempo realization, and cue recognition.

IV. EXPERIMENTS

  • The evaluation covers physical tempo compliance, task competence, behavior without tempo cues, transfer to unseen expressions, component contributions, and real-robot validation.Source: §IV · p. 5

A. Experimental Setup and Evaluation Metrics

B. Tempo Control under Canonical Instructions

Under canonical quickly/slowly instructions, TempoBridge substantially improves tempo compliance while keeping task success comparable to the baseline. It also largely preserves task competence without tempo cues, and qualitative trajectories illustrate execution under canonical and unseen expressions.

B. Tempo Control under Canonical Instructions

C. Task Competence without Tempo Cues

D. Generalization to Unseen Tempo Expressions

TempoBridge retains substantial tempo control for expressions not used to construct the readout, though recognition varies by expression. Component-wise diagnostics report readout accuracy, phase agreement, and realized speed separation for canonical and unseen cues.

D. Generalization to Unseen Tempo Expressions

E. Component-wise Analysis

  • Tempo Success Rate only requires FAST speed to exceed SLOW speed, whereas the reported median speed ratio quantifies the magnitude of that separation.Source: E. Component-wise Analysis · p. 7
  • Readout Accuracy is 97.5% for canonical and 78.4% for unseen instructions, while Balanced Phase Agreement is 89.7% and 90.4%, respectively.Source: E. Component-wise Analysis · p. 7, Table V · p. 7
  • Median FAST-to-SLOW speed ratios are 1.58× for canonical and 1.52× for unseen instructions, indicating substantial realized speed differences beyond the binary tempo criterion.Source: E. Component-wise Analysis · p. 7

F. Real-World Tempo Control

Across three real-world tasks, TempoBridge changed TCP speed according to FAST/SLOW instructions while achieving task success close to the task-adapted baseline. The conclusion notes that the framework currently handles discrete tempos and relatively simple task phases.

F. Real-World Tempo Control

  • Both real-robot methods used the same task-specific checkpoints and execution settings, without tempo-specific policy fine-tuning.Source: F. Real-World Tempo Control · p. 7
  • TempoBridge achieved 60 successful executions out of 67 attempts (89.6%), versus the baseline’s 60 out of 66 (90.9%); quickly segments had higher median TCP speed, with cue reversal reversing the ordering.The figure summarizes 10 successful rollouts per condition; across the two segments, speed is measured over the central 50% of each segment’s duration.Source: F. Real-World Tempo Control · p. 7, Figure 5 · p. 8

V. CONCLUSIONS

  • Across 40 LIBERO tasks, TempoBridge improved tempo compliance while largely preserving task competence, maintaining near-baseline performance without tempo cues, and generalizing to unseen tempo expressions.Source: §V · p. 8
  • The framework currently focuses on discrete FAST/SLOW control and relatively simple task-phase structures.Source: §V · p. 8
Loading 2610.09451v1…