Source-linked AI summary
Learning Native Continuation for Action Chunking Flow Policies
Yufeng Liu, Hang Yu, Juntu Zhao, Bocheng Li, Di Zhang, Mingzhu Li, Wenxuan Wu, Yingdong Hu, Junyuan Xie, Junliang Guo, Dequan Wang, Yang Gao
TL;DR
Naive chunked VLA execution can produce discontinuities, while RTC addresses continuity externally and may leave spurious multimodal switching. Legato learns schedule-shaped, per-step continuation dynamics with randomized schedule conditioning, and real-world experiments show consistent improvements over RTC across five manipulation tasks.
Problem
Naive action chunking produces discontinuities at chunk boundaries, while RTC's inference-time continuation is not learned by the policy and can leave spurious multimodal switching.
Method
Legato reshapes flow dynamics for schedule-shaped per-step continuation and conditions the policy on randomized schedules to support varying inference delays and smoothness control.
Results
Legato consistently outperforms RTC across five real-world manipulation tasks, producing smoother trajectories and shorter task completion times.
Takeaways & Limitations
Learning continuation within the policy reduces spurious multimodal switching and supports more consistent action modes, less hesitation, and shorter task completion time.
Abstract
from arXiv · showhide
Action chunking enables Vision Language Action (VLA) models to run in real time, but naive chunked execution often exhibits discontinuities at chunk boundaries. Real-Time Chunking (RTC) alleviates this issue but is external to the policy, leading to spurious multimodal switching and trajectories that are not intrinsically smooth. We propose Legato, a training-time continuation method for action-chunked flow-based VLA policies. Specifically, Legato initializes denoising from a schedule-shaped mixture of known actions and noise, exposing the model to partial action information. Moreover, Legato reshapes the learned flow dynamics to ensure that the denoising process remains consistent between training and inference under per-step guidance. Legato further uses randomized schedule condition during training to support varying inference delays and achieve controllable smoothness. Empirically, Legato produces smoother trajectories and reduces spurious multimodal switching during execution, leading to less hesitation and shorter task completion time. Extensive real-world experiments show that Legato consistently outperforms RTC across five manipulation tasks, achieving approximately 10% improvements in both trajectory smoothness and task completion time.
I. INTRODUCTION
Legato makes chunk continuation a learned property of flow-based VLA policies, addressing discontinuities and multimodal switching at chunk boundaries. Its schedule-shaped, per-step guidance and randomized conditioning improve smoothness and efficiency across five real-world manipulation tasks.
- Naive action chunking can create discontinuous transitions because of inference delay and flow-policy multimodality.
- RTC improves continuity through inference-time inpainting but leaves continuation outside the policy, allowing spurious switching, hesitation, and abrupt direction changes.
- Legato reshapes the learned flow dynamics so chunk continuation is native to the policy and remains consistent between training and inference under per-step guidance.
- Its smooth guidance schedule anchors the chunk prefix to known actions, then ramps guidance strength down to zero for fine-grained continuity control.
- Randomized schedule conditioning supports varying inference delays and lets one policy adapt continuation strength without retraining.
- Across five manipulation tasks, Legato consistently produces smoother trajectories and shorter task completion times than RTC while suppressing spurious multimodal switching.
C. Conditioning in Diffusion- and Flow-Based Policies
Flow-based VLA policies generate fixed-length action chunks by transporting noise to actions, while RTC applies inference-time guidance to improve continuity. One-shot prefix guidance fails to preserve overlap constraints during denoising, motivating repeated guidance and Legato’s training-inference alignment.
- Flow matching: Flow matching transforms Gaussian noise into fixed-length action chunks through a learned velocity field.At inference, generation integrates the velocity field across denoising time.
- Real-Time Chunking: RTC improves continuity by constraining overlapping regions of newly generated chunks to previously generated actions.Its structured schedule applies full guidance near the boundary before gradually relaxing the constraint.
- Why per-step guidance matters: One-shot prefix guidance cannot preserve overlap constraints because the guided actions drift from their reference during multi-step denoising.The drift occurs despite clamping the prefix at initialization.
- Legato: Legato reshapes flow dynamics during training so per-step, schedule-shaped guidance remains consistent with the training objective.The method targets a native continuation property rather than relying solely on inference-time correction.
C. Native Continuation for Action Chunk Generation
Legato makes continuation a native property of the policy by combining schedule-shaped action-noise mixtures with velocity-field reshaping. The resulting per-step dynamics preserve training-inference consistency while suppressing mode switching and boundary discontinuities.
- Native continuation: Legato requires per-step guidance and training-inference consistency for native action continuation.These requirements motivate the schedule-shaped training path and induced guided dynamics.
- Action-noise mixture: The training path mixes reference actions with noise according to a horizon-wise guidance schedule ω.ω provides full guidance near the chunk beginning and ramps toward weaker guidance later.
- Multimodal persistence and smoothness: Large guidance weights suppress effective transport, making overlap and ramp regions less mutable during denoising.The schedule discourages frequent switching among competing modes and gradually relaxes continuation toward free generation.
- Training-inference consistency: Legato reshapes the velocity field so the network regresses toward a target consistent with repeated continuation guidance.This preserves the geometric direction of standard flow matching while modifying velocity magnitude.
- Inference: At inference, Legato uses the previous predicted chunk as the continuation reference and integrates the learned dynamics with the training discretization.The reference is aligned into Aref before repeated guidance and denoising updates.
D. Schedule Randomization and Conditioning
Legato represents continuation with a schedule determined by inference delay and ramp length, then randomizes and exposes that schedule during training. This supports changing continuation regimes without retraining.
- Schedule parameters: The continuation schedule is uniquely specified by inference delay d and ramp length r.It consists of a full-guidance prefix followed by a ramp section.
- Schedule randomization: Randomizing d and r during training exposes the policy to diverse guidance schedules for variable latency and controllable smoothness.The schedule family addresses differences across hardware platforms and inference optimizations.
- Schedule conditioning: Schedule conditioning lets inference adapt continuation behavior by changing ω without retraining the model.The action decoder receives the schedule as an additional feature, improving robustness across hardware platforms and inference budgets.
IV. EXPERIMENTS
The real-world evaluation compares RTC and Legato across five manipulation tasks using task success, completion time, and multiple smoothness measures. The experiments use controlled training conditions and include both global and chunk-overlap continuity metrics.
- Evaluation metrics: Smoothness is assessed with NLDLJ, NSPARC, and overlap RMSE, for which lower values indicate better performance.NLDLJ captures global jerk-related smoothness, while NSPARC measures frequency-domain velocity-profile smoothness.
- Tasks and environments: The evaluation covers five tasks: stacking bowls, pouring, placing items into a box, folding a towel, and opening a drawer.The tasks span rotation- and translation-dominant motions and multimodal grasp or arm selection.
- Evaluation metrics: The study measures task completion score, completion time, trajectory smoothness, and chunk-overlap RMSE.Completion time captures delays associated with hesitation or spurious action switching, while overlap RMSE evaluates local continuity at chunk connections.
- Models and training protocol: RTC and Legato are compared under controlled conditions using the same pretrained checkpoint, datasets, hyperparameters, and training-step count.This protocol is intended to isolate differences between the two methods.
B. Main Results
Across five real-world manipulation tasks, Legato consistently outperforms RTC in efficiency, trajectory smoothness, and task success. Its schedule parameters expose a trade-off between overlap consistency and global frequency-domain smoothness.
- Overall results: Legato consistently outperforms RTC across five physical-robot manipulation tasks.The evaluation reports improvements across task efficiency, smoothness, and success.
- Task efficiency: Legato achieves shorter completion times by suppressing multimodal oscillations and reducing hesitation during execution.The effect is especially pronounced when visually similar objects create competing action modes.
- Trajectory smoothness: Legato improves smoothness metrics, with statistically significant gains except for NLDLJ.NSPARC and chunk-overlap RMSE improve, while NLDLJ does not consistently reach significance but does not degrade.
- Task success: Legato exceeds RTC in task completion scores without sacrificing execution efficiency or trajectory smoothness.The reported task-success improvement supports more reliable real-world manipulation performance.
- Schedule trade-offs: Reducing stride strengthens overlap consistency but can degrade global smoothness, while shortening the ramp improves frequency-domain smoothness at weaker overlap alignment.Schedule configurations therefore provide controllable smoothness trade-offs.
C. Comparison with Training-Time RTC
Legato differs from training-time RTC by reshaping policy dynamics rather than applying hard overlap constraints alone. Ablations show robustness across schedules, models, smoothing methods, and denoising conditions, while exposing a consistency–smoothness trade-off.
- Training-time RTC comparison: Legato achieves higher task scores, shorter completion times, and improved smoothness metrics than training-time RTC on the pour task.Both methods are compared under the same experimental settings.
- Method comparison: Legato reshapes learned flow dynamics, whereas training-time RTC uses hard prefix conditioning while leaving the underlying flow dynamics unchanged.A zero-length ramp only makes the constraint shapes similar; the learned continuation mechanisms remain fundamentally different.
- Schedule robustness: Legato consistently outperforms RTC across schedule configurations and remains superior as stride and delay vary.Execution stride and inference delay act as control knobs for trajectory smoothness.
- Ablation studies: Reducing execution stride improves chunk-to-chunk consistency but can degrade whole-trajectory smoothness through accumulated high-frequency components.Shortening the ramp with a small stride reverses this balance by improving NSPARC while weakening overlap consistency.
- Conditioning: Explicit schedule conditioning improves adaptation to different continuation regimes induced by varying delay and ramp parameters.Removing the condition row degrades trajectory smoothness and execution stability.
E. Simulation Results on Kinetix
In Kinetix, Legato generalizes beyond the main policy setting and outperforms competing methods under positive inference delay. Schedule randomization is important for performance across delays, while the current denoising formulation limits inference-time adjustment.
- Kinetix evaluation: Legato outperforms all compared methods in Kinetix when inference delay is greater than zero.The experiment uses settings identical to Training-time RTC and a UNet-based policy.
- Schedule randomization: Fixing delay during training degrades performance at other evaluation delays, highlighting the importance of delay randomization.Excluding zero delay from the randomized range can cause near-complete evaluation failure at the first unguided denoising step.
- Deployment: The deployment recommendation uses the maximum recent delay, half-chunk execution stride, and the remaining horizon as ramp length.For variable denoising-step inference, the no-condition-row variant is recommended.
- Training: Initializing Legato from a strong pretrained base policy yields more stable convergence and reduces excessive-guidance impairment of reactivity.The recommendation is based on observed training behavior rather than a new benchmark comparison.
- Limitation: The current formulation specifies the denoising step at training time, limiting adjustment during inference.More flexible native continuation schemes are identified as future work.
APPENDIX
The appendix describes evaluation protocols, task-completion scoring, and three complementary trajectory-smoothness metrics used across five real-world manipulation tasks.
- Evaluation protocol: Five manipulation tasks use shared initial configurations and, unless noted, randomized object conditions held identical across models.The tasks include pouring, pick-and-place, drawer opening, and towel folding; the supplied protocol also states evaluation counts unless otherwise specified.
- Evaluation protocol: The guidance schedule is uniquely determined by inference delay d and ramp length r, with experiments controlling delay explicitly.Main experiments use d = 8 timesteps, while the delay ablation evaluates d ∈ {6, 8, 10}.
- Task completion scoring: Task completion scores provide graded feedback by rewarding meaningful intermediate steps and penalizing execution errors.Task-specific criteria and penalty rules are defined for all five tasks rather than relying only on binary success.
- Smoothness metrics: Three complementary smoothness metrics—NSPARC, NLDLJ, and overlap RMSE—are reported so that smaller values indicate smoother trajectories.NSPARC measures frequency-domain spectral smoothness, NLDLJ penalizes rapid acceleration changes, and overlap RMSE measures inter-chunk consistency.
E. Results on More Challenging Tasks
Legato improves performance on fine-manipulation tasks where jitter can cause failure, with advantages in execution time, smoothness, and success rates.
- Legato improves time, smoothness, and success rates on the Skewer Fruit and Insert Pen tasks.Both tasks require fine manipulation, making them sensitive to jitter; evaluations used 20 trials per method.
F. Experiment on the Smoothness–Reactivity Trade-off
Legato preserves reactivity while improving trajectory smoothness, and repeated per-step guidance better maintains continuation constraints than one-shot initialization guidance.
- Smoothness–reactivity trade-off: Legato achieves similar reactivity to RTC while significantly improving smoothness in a reactive ball-pulling task.The task pulls the ball perpendicular to the gripper’s motion at fixed speed, enabling a direct comparison of reactivity and smoothness.
- Guidance ablation: Legato significantly outperforms one-shot guidance, particularly on overlap RMSE, under matched pour-task conditions.The comparison uses stride s = 30, delay d = 8, and ramp length r = 22.
- Guidance ablation: Repeated guidance before every denoising step is necessary to maintain consistent continuation across action chunks.Initialization-only guidance cannot reliably preserve constraints throughout denoising.
- Trajectory behavior: RTC exhibits low-frequency, large-amplitude oscillations and more spurious multimodal switching, whereas Legato produces smoother pour trajectories.The trajectory example shows direction changes occurring mostly within RTC chunks despite lower overlap RMSE.
H. Robot Hardware Configuration
The experiments use a dual-arm, three-camera robotic platform with 14-dimensional end-effector actions, while appendix analyses examine inference frequency, oscillations, continuation metrics, and guidance conditions.
- Robot Hardware Configuration: The platform has two arms, each with seven actuated joints and a gripper, for eight degrees of freedom per arm.
- Robot Hardware Configuration: Three RGB cameras provide visual observations: one head-mounted global camera and one wrist-mounted camera per arm.
- Robot Hardware Configuration: Each arm uses a 7-dimensional end-effector action, giving the dual-arm system a 14-dimensional action space.The action combines a 6-dimensional pose with a 1-dimensional gripper command per arm.
- Inference-frequency analysis: When delay d is fixed, setting s=r=d gives the highest possible inference frequency because the next inference starts immediately after each chunk.
- Trajectory smoothness analysis: RTC shows large-amplitude oscillations in the highest-frequency regime, while Legato produces substantially smoother motion and suppresses low-frequency oscillatory behavior.Legato’s stronger mode persistence across denoising steps is associated with this suppression in the reported analysis.
- Metric limitation: Overlap RMSE can miss smoothness degradation when trajectories contain low-frequency, large-amplitude oscillations.RTC achieves lower overlap RMSE than Legato in the analyzed configuration despite visibly worse oscillatory motion.
- Condition-row analysis: Adding the condition row consistently reduces overlap RMSE but does not significantly improve NSPARC, indicating stronger inter-chunk consistency than intra-chunk smoothness.Smaller delays can also reduce overlap RMSE without the condition row, and omitting it may remain viable when d is sufficiently small.