Real-Time Robot Execution with Masked Action Chunking

Haoxuan Wang; Gengyu Zhang; Yan Yan; Yuzhang Shang; Ramana Rao Kompella; Gaowen Liu

Source-linked AI summary

Real-Time Robot Execution with Masked Action Chunking

Haoxuan Wang, Gengyu Zhang, Yan Yan, Yuzhang Shang, Ramana Rao Kompella, Gaowen Liu

arXiv:2601.20130v1Jan 27, 2026cs.RO

TL;DR

Real-time robot manipulation under asynchronous inference is challenged by inter-chunk discontinuity and intra-chunk inconsistency, where executed actions can misalign with current perception. REMAC learns corrective adjustments through masked action chunking and uses prefix-preserved sampling to improve continuity without additional inference delay. Experiments in simulation and real-world settings report faster task completion, robustness to varying delays, and higher completion rates.

Problem
Asynchronous inference enables real-time execution but can fail when inherited actions become misaligned with current observations, creating intra-chunk inconsistency alongside inter-chunk discontinuity.
Method
REMAC learns corrective adjustments on a pretrained policy through masked action chunking and uses previously executed actions as priors in a prefix-preserved sampling pipeline.
Results
Across simulation and real-world benchmarks, the method is robust to varying delays and achieves faster task completion, higher success rates, and no additional inference delay.
Takeaways & Limitations
The method provides a stronger policy for asynchronous execution and can be combined with existing test-time algorithms.
Takeaways & Limitations
The method requires specifying a maximum inference delay in advance; exceeding that bound may cause unexpected failure, and masked finetuning may require substantial data.

Abstract

from arXiv · show

Real-time execution is essential for cyber-physical systems such as robots. These systems operate in dynamic real-world environments where even small delays can undermine responsiveness and compromise performance. Asynchronous inference has recently emerged as a system-level paradigm for real-time robot manipulation, enabling the next action chunk to be predicted while the current one is being executed. While this approach achieves real-time responsiveness, naive integration often results in execution failure. Previous methods attributed this failure to inter-chunk discontinuity and developed test-time algorithms to smooth chunk boundaries. In contrast, we identify another critical yet overlooked factor: intra-chunk inconsistency, where the robot's executed action chunk partially misaligns with its current perception. To address this, we propose REMAC, which learns corrective adjustments on the pretrained policy through masked action chunking, enabling the policy to remain resilient under mismatches between intended actions and actual execution during asynchronous inference. In addition, we introduce a prefix-preserved sampling procedure to reinforce inter-chunk continuity. Overall, our method delivers more reliable policies without incurring additional latency. Extensive experiments in both simulation and real-world settings demonstrate that our method enables faster task execution, maintains robustness across varying delays, and consistently achieves higher completion rates.

1 INTRODUCTION

Real-time robot manipulation requires continuous executable actions, but inference latency and action-chunk boundaries make synchronous execution impractical and asynchronous execution vulnerable to discontinuities and perception–action mismatch. REMAC addresses both inter-chunk continuity and intra-chunk consistency through masked action chunking and refined sampling, without added inference delay.

Real-time robot control must maintain a continuous stream of executable actions so the robot never runs idle.
Action chunking improves real-time control but reduces reactivity to sudden state changes and introduces discontinuities at chunk boundaries.
Asynchronous inference predicts upcoming actions while current actions execute, enabling real-time execution but worsening inter-chunk discontinuity and introducing intra-chunk inconsistency.
Prior methods mainly refined upcoming chunks at test time to mitigate inter-chunk discontinuity, using executed actions as informative priors.
REMAC learns corrective adjustments by masking arbitrary portions of action chunks, while its sampling pipeline incorporates previously executed actions as priors to improve inter-chunk continuity.
Experiments across 12 simulated tasks and three real-world settings show higher success rates, faster task completion, smoother dynamics, and no additional inference delay.

2 RELATED WORKS

VLA policies and action chunks support generalist, temporally extended robot behavior, but chunk boundaries can create discontinuities and distribution shifts. Prior work therefore seeks smoother transitions while balancing long-term consistency with short-term reactivity.

VLA models translate human instructions and sensory observations into physical actions and are a leading approach toward generalist policies across tasks, environments, and robot embodiments.
Action chunking provides temporal abstraction by generating coherent behavior segments that a low-level controller executes.
Figure 1 contrasts sequential synchronous inference with concurrent asynchronous inference, which enables real-time execution but introduces inter-chunk discontinuity and intra-chunk inconsistency.
Chunk boundaries can introduce discontinuities and distribution shifts even when actions remain temporally consistent within a segment.
Temporal Ensembling aggregates overlapping actions, while Bidirectional Decoding addresses the balance between long-term consistency and short-term reactivity.

3 PRELIMINARIES

The paper formalizes chunked policy execution and inference delay, then contrasts synchronous and asynchronous inference. Asynchrony keeps actions available for real-time control but creates boundary discontinuity and mismatch between current observations and inherited actions.

The policy predicts an action chunk At of length P from observation ot, but rollout executes only the first h actions, where 1 ≤ h ≤ P.
Inference delay is the lag between acquiring ot and making At available, discretized as d := ⌊δ/∆t⌋ from latency δ and controller period ∆t.
Flow-matching policies generate chunks by sampling an initial latent action sequence and integrating a learned velocity field over normalized time τ ∈ [0, 1].
Synchronous inference waits for the current execution horizon before predicting the next chunk, requiring δ < ∆t, or d = 0, for real-time execution.
With 50 Hz control, π0 action generation alone requires 76 ms on an NVIDIA RTX 4090, making synchronous execution increasingly constrained as control frequencies and model sizes rise.
Asynchronous inference keeps actions available while prediction runs concurrently, but can create incoherent chunk boundaries when consecutive trajectories diverge after a shared history.
Under delay, the first actions may come from the previous chunk while later actions come from the current chunk, making inherited actions suboptimal for the current state.

4 METHODOLOGY

REMAC adapts a pretrained action-chunking policy to asynchronous execution by training it on delay-conditioned masked chunks and self-conditioned inputs. Prefix-preserved sampling then maintains executed actions while synthesizing only the remaining chunk segment.

Masked Action Chunking: REMAC learns a delay-aware policy that addresses intra-chunk inconsistency by adapting the pretrained policy to mismatches between observations and executed actions.The method is designed for asynchronous inference, where execution of previous-chunk actions can misalign with the current observation.
Masked Action Chunking: Prefix masking restricts supervision to the executable portion of each action chunk while excluding the already-committed prefix.Inference delays are sampled across valid values so training exposes the policy to varying masking conditions.
Masked Action Chunking: Self-conditioned curriculum training gradually replaces ground-truth action inputs with pretrained-policy predictions, aligning training inputs with test-time conditions.The mixing schedule anneals from pure ground-truth input to pure self-conditioned input, with stop-gradient applied to pretrained predictions.
Training and Adaptation: Training uses a LoRA-adapted target policy and updates it with the combined masking and residual-alignment objective across sampled delays.The training loop samples delays, constructs masked inputs, computes target-policy and pretrained-policy flows, and minimizes the combined loss.
Masked Action Chunking: Residual alignment adds an objective that matches the corrective adjustment to the residual between pretrained predictions and ground-truth targets.This complements direct ground-truth supervision by explicitly modeling the correction relative to the pretrained policy.
Prefix-Preserved Sampling: Prefix-preserved sampling initializes from prior actions and keeps the overlapping executed segment fixed while newly synthesizing only the remaining chunk.At rollout initialization, the absence of prior actions is represented with zero or random Gaussian initialization before standard integration.

5 EXPERIMENTS

Experiments evaluate REMAC against baseline methods across simulated and real-world tasks under varied inference delays. REMAC consistently improves completion, efficiency, robustness, and trajectory smoothness, while ablations support the contribution of its components.

Experimental scope: Experiments span 12 simulated tasks and three real-world settings, comparing REMAC with asynchronous, synchronous, and test-time baselines under varied delays.The evaluation includes component ablations and delay-injection studies.
Simulation results: REMAC consistently outperforms all baselines across delay settings, with especially pronounced gains under larger inference delays.Performance is measured using task success rate and completion time in simulation.
Simulation results: REMAC achieves shorter average execution time and a smaller performance drop and lower execution-time increase as inference delay rises.The method also improves performance when d = 0.
Ablations: The full method achieves the highest overall success rate, while the curriculum schedule improves performance and training stability over pure ground-truth or fully self-conditioned inputs.Adding LoRA alone produces no performance gain, whereas progressively adding the proposed components improves results.
Real-world results: In real-world tasks, REMAC achieves higher completion rates across all tasks and produces smoother, faster execution under a 150ms injected delay.The comparison uses task progress and average robot kinematics over 15 trials.
Real-world results: Delay injections of 75ms and 150ms show that REMAC remains superior under varying latency, while RTC exhibits significantly degraded performance.The injections simulate slower hardware and network conditions.

6 CONCLUSION

The paper addresses inter-chunk discontinuity and intra-chunk inconsistency in asynchronous robot manipulation with REMAC. It combines masked action chunking and prefix-preserved sampling without adding inference delay.

Conclusion: REMAC targets exacerbated inter-chunk discontinuity and intra-chunk inconsistency in asynchronous inference.The method addresses both mismatches between chunks and mismatches between observations and executed actions.
Conclusion: REMAC learns corrective adjustments on a pretrained policy through masked action chunking and reinforces continuity through prefix-preserved sampling.The approach is designed as an integration with existing VLA frameworks.
Conclusion: The method introduces no additional inference delay compared with the pretrained policy.This preserves the real-time execution setting targeted by the paper.

C TASK EXAMPLES

The task examples show the wrist-camera perspectives used for policy input and the range of manipulation difficulty represented in the experiments.

Task examples: The task examples present wrist-camera views used as policy inputs during the experiments.These views illustrate the visual perspective available for manipulation.
Task examples: The examples span varying levels of manipulation difficulty across the included tasks.The examples are intended to show the visual and task diversity of the evaluation.

D PER-TASK SIMULATION RESULTS

Per-task ablation results show consistent success-rate improvements from the components of REMAC, supporting the generalizability of the design choices across tasks.

Per-task simulation results: Per-task ablations show that each REMAC component contributes consistent improvements in success rate.The results correspond to the component comparisons reported in Table 1.
Per-task simulation results: The per-task pattern supports the generalizability of the proposed design choices.Improvements are reported across the evaluated tasks rather than for only one task.

E ADDITIONAL ABLATIONS

REMAC remains relatively insensitive to the qmax and qmin choices, while learned mask embeddings produce mostly stable performance with only marginal task-level changes.

E.1 EFFECT OF qMAX AND qMIN: Performance varies only slightly across qmax and qmin settings, with larger values—especially qmin—tending to worsen results.The paper attributes this to always masking a longer prefix when qmin is larger, encouraging over-conservative rollouts.
E.1 EFFECT OF qMAX AND qMIN: The adopted setting is qmax = 4 and qmin = 0, but these hyperparameters do not limit the inference-delay range REMAC can handle.They control the strength of mask-induced training perturbations rather than the delays supported at deployment.
E.2 EFFECT OF LEARNING A MASK EMBEDDING: Adding a learned mask embedding yields mixed but generally stable outcomes, with most tasks retaining similar performance.The mask is projected into a learnable embedding and injected as an additional model signal.
E.2 EFFECT OF LEARNING A MASK EMBEDDING: REMAC does not depend critically on whether the delay mask is embedded or used directly.This indicates robustness to the tested architectural variation under the same training setup.

E.3 DATA-EFFICIENCY OF REMAC

Additional experiments examine REMAC under reduced data, distribution changes, unseen language, and alternative policy architectures. The results show mild low-data degradation, preserved supported generalization, and compatibility beyond flow-matching policies.

E.3 DATA-EFFICIENCY OF REMAC: With only 10 demonstrations per task instead of 200, REMAC remains close to the full-data model with only mild degradation.The paper attributes this low-data behavior to self-supervised masking that exposes the model to diverse prefix deviations without extra demonstrations.
E.4 EFFECT OF REMAC ON GENERALIZABILITY: Under scene and background variation, REMAC behaves similarly to the baseline and reliably grounds objects seen during fine-tuning.Both policies remain effective despite altered curtains, table covering, and added distractors.
E.4 EFFECT OF REMAC ON GENERALIZABILITY: Under unseen language instructions, both the baseline and REMAC fail on novel tasks or unseen objects, indicating the limitation lies in the underlying fine-tuned policy.The tested examples include unseen objects such as “cup” and “box.”
E.4 EFFECT OF REMAC ON GENERALIZABILITY: On fine-tuned π0.5, REMAC preserves recognition of unseen objects and does not noticeably degrade the model’s existing out-of-distribution capability.The underlying π0.5 still struggles with unseen tasks under unseen language prompts.
E.4 EFFECT OF REMAC ON GENERALIZABILITY: REMAC leaves VLM grounding and perception capabilities untouched by applying low-rank adjustments only to the action expert.This design is presented as the reason it does not diminish the fine-tuned base model’s generalizability.
E.5 GENERALITY ACROSS POLICY ARCHITECTURES: REMAC transfers beyond flow-matching policies to Transformer-based chunking policies, including ACT.In ACT experiments, LoRA is applied to decoder layers and the action head, with ACT-specific ℓ1 and KL objectives.

F ROBUSTNESS TO VARYING DELAY CONDITIONS

REMAC is evaluated under noisy, fluctuating, spiky, and corrupted delays in simulation and real-world settings. It degrades gracefully, while delay discretization and the tested controller period provide additional robustness to moderate latency variation.

F ROBUSTNESS TO VARYING DELAY CONDITIONS: The robustness evaluation covers noisy, rapidly fluctuating, and adversarially spiky delay patterns in simulation and the real world.Real-world delays combine inference, communication, file I/O, memory contention, and scheduling-jitter effects.
F ROBUSTNESS TO VARYING DELAY CONDITIONS: In simulation, delays are sampled from {d − 1, d, d + 1} or replaced by the maximum valid value with 10% probability.The first scheme makes 66% of sampled delays inaccurate, while the second models infrequent large latency spikes.
F ROBUSTNESS TO VARYING DELAY CONDITIONS: REMAC under corrupted delays still outperforms RTC under accurate delays, while both methods degrade gracefully without catastrophic failures.This result is reported over 12 Kinetix tasks under extremely ill-conditioned latency sequences.
F ROBUSTNESS TO VARYING DELAY CONDITIONS: Under real-world delay corruption sampled from {d−1, d, d+1, d+2}, the system incurs only 1–2 additional failures out of 10 trials.The failures primarily result from d+2 overestimation causing temporary pauses and exceeding the 300-step time limit.
F ROBUSTNESS TO VARYING DELAY CONDITIONS: The paper characterizes these extra failures as a timeout artifact rather than evidence of policy instability.This qualification specifically applies to the tested corrupted-delay real-world stress test.
F ROBUSTNESS TO VARYING DELAY CONDITIONS: With a 67 ms controller sampling period at 15 Hz, a one-step delay change requires continuous latency to shift by more than 67 ms.This discretization smooths moderate noise in continuous inference latency.

G LIMITATIONS

REMAC has two stated limitations: it requires a prespecified maximum inference delay, and masked fine-tuning may require substantial data when collection is costly or constrained.

G LIMITATIONS: If execution delay exceeds the prespecified maximum, unexpected failure may occur.The maximum delay is required so optimization covers the full range of possible delays.
G LIMITATIONS: Masked fine-tuning may demand substantial data, limiting practicality when data collection is costly or constrained.This is identified as a separate limitation from the maximum-delay requirement.