Robotic Spatiotemporal Editing: A Unified Interface for Controllable Action Generation via Masked Action Reconstruction

1 INTRODUCTION

• Robots must satisfy constraints on where, when, and how actions execute, while available references often specify only partial future motion.Examples include keypoints, flow, keyposes, and grasp proposals.Source: §1 · p. 2
• MAP represents heterogeneous references as partially specified temporal Cartesian actions, and IP reconstructs masked components into complete executable trajectories.The interface is intended to support unseen combinations of control references.Source: Figure 1 · p. 1, §1 · p. 2
• Inpainting Policy supports diverse spatiotemporal references, with 44.3 mm mean position error, 7.8° mean rotation error, and a 4.67-percentage-point LIBERO-10 success-rate improvement over baseline.The reported references include keypoints, flows, orientations, poses, and combinations of these types.Source: §1 · p. 2
• MAP provides a human-readable, editable interface that indicates which timesteps and action dimensions are constrained.Source: §1 · p. 2
• A 49M-parameter DiT achieves 92.67% success rate and 0.023 control error in ManiSkill, while a 118M-parameter diffusion model achieves above 90% success in real-world experiments.Source: §1 · p. 2

2.1 HETEROGENEOUS SPATIOTEMPORAL SIGNALS FOR ROBOT CONTROL

• Prior spatial-control approaches include sparse keypoints, keyposes, grasp proposals, affordances, trajectories, and video-based prediction to guide manipulation.Examples include KALM, BiKC, AnyGrasp, Gemini Robotics-ER, RoboBrain, A0, V-JEPA 2-AC, and UVA.Source: §2.1 · p. 3
• The shared Cartesian action representation lets one policy reconstruct remaining action entries across different reference types and combinations.Source: §2.1 · p. 3

2.2 INPAINTING FOR ROBOT CONTROL

• Inpainting has been used for visual augmentation and masking, motion-feature completion and inbetweening, action-prefix preservation, and trajectory-endpoint or semantic-keyframe constraints.Source: §2.2 · p. 3

3.1 MASKED ACTION SPACE

• MAP converts heterogeneous spatiotemporal references into specified action tokens and binary masks, representing them in a shared Cartesian action space.Structured references include keypoints, flow, keyposes, orientations, and gripper states; mixed and unstructured constraints are also supported.Source: §3.1 · pp. 3–4

3.2 PROGRESS-GUIDED SLIDING WINDOW OF MAP

• Normalized task progress aligns references to the current task stage because execution speed can vary across rollouts, making raw timesteps unreliable.The estimated progress selects a local reference window with past and future context for action generation.Source: §3.2 · p. 4

3.3 INPAINTING POLICY

• The Inpainting Policy generates complete Cartesian action chunks from visual observations, robot state, and the local MAP window by reconstructing masked action tokens.Its weighted reconstruction objective balances known and unknown regions across training examples, sampled masks, diffusion steps, and Gaussian noise.Source: §3.3 · p. 5

3.4 MULTIMODAL JOINT TRAINING

• Joint training uses structured reference masks, no-reference examples, and random token masks; the trained model follows constraint compositions and generalizes to an unseen OOD mask pattern.Random-mask training may underperform structured joint training when training data is limited.Source: §3.4 · p. 5

3.5 INPAINTING VIA REPAINT

• The training-free RePaint strategy reinserts specified reference values during denoising, and its final step overwrites known tokens with clean values to preserve them exactly.At each reverse step, the method replaces known entries using references noised to the corresponding diffusion level.Source: §3.5 · pp. 5–6

4 EXPERIMENTS

• The evaluation spans 15 tasks across ManiSkill, LIBERO-10, and real-world settings, using masked demonstrations to construct diverse simulation reference conditions without additional annotations.Each demonstration is paired with multiple mask configurations; the LIBERO-10 policy is trained first with random masks and then with structured masks.Source: §4 · p. 6, §4.2 · p. 8

4.1 STUDY ON ACTION INPAINTING

• For structured 3D-point masks, success rises from 57.0% at visible ratio 0.02 to 91.7% at 0.50, while random masks reach 71.7% at the highest ratio.The random-mask result is less stable; the study varies visible ratio and compares structured and random constraints.Source: §4.1 · p. 7
• At approximately equal visible-token counts, Inpainting Policy outperforms the unconditioned DiT policy across mask types; positional and full-pose constraints guide task completion more strongly than orientation-only constraints.Semantically structured references are more effective than an equivalent number of arbitrary visible action tokens.Source: §4.1 · p. 7, Figure 4 · p. 7

4.2 UNIFIED CONTROL VIA JOINT TRAINING

• Inpainting Policy reaches average success rates of 64.67% on LIBERO-10 and 92.67% on ManiSkill, improving task completion while lowering control error; RePaint further reduces CE with a control-accuracy/task-completion trade-off.The table evaluates checkpoints across reference types using success rate and control error; LIBERO-10 uses a shared multitask policy, while ManiSkill uses separate task policies.Source: §4.2 · p. 8, Table 1 · p. 8
• For the unseen Mix0 mask, Inpainting Policy achieves 80.56% success, while its control error falls from approximately 0.049 to 0.022 during training.Mix0 combines mask patterns absent from training; RePaint maintains CE around 0.014–0.015 across checkpoints.Source: §4.2 · pp. 8–9

4.3 CONTROL WITH REAL-WORLD REFERENCES

• With three or more real-world reference points, success exceeds 93%; average cup-to-reference distance is 44.3 mm, and mean tube-tilt error is 7.8°.With only one reference at the final progress point, task completion falls below 30%.Source: §4.3 · pp. 9–10

5 CONCLUSION AND FUTURE WORK

• The framework adds a unified control interface to DiT- or U-Net-based policies, enabling users to edit and compose spatiotemporal constraints through masked action inpainting without extra control-specific data or backbone training budget.The authors state that the method preserves original task completion ability.Source: §5 · p. 10
• The inpainting formulation may be compatible with foundation action models such as WAM and flow-matching-based VLAs, but this remains for future empirical study.Source: §5 · p. 10
• The current inpainting implementation may not be optimal; CFG and ControlNet-style conditioning remain unexplored, as do large-scale BERT-style random-mask pretraining and zero-shot editing under less structured controls.Source: §5 · p. 10

AI USE DISCLOSURE

• Generative AI tools assisted with translating and polishing the manuscript, drafting and editing, improving readability, modifying figures, interpreting results, and writing or modifying code.Source: AI USE DISCLOSURE · p. 11
• The tools were not used to generate synthetic datasets, develop theoretical models or conceptual frameworks, formulate or prove mathematical claims, design methodology or experiments, clean datasets, or perform qualitative or thematic analysis.Source: AI USE DISCLOSURE · p. 11
• The authors state that they reviewed and verified all AI-assisted content and take full responsibility for the submission’s text, claims, results, code, and figures.Source: AI USE DISCLOSURE · p. 11

ETHICS STATEMENT

• The study concerns controllable robot manipulation using public simulation benchmarks and controlled real-world robotic experiments.Source: ETHICS STATEMENT · p. 11
• The work does not involve human subjects, personal data, sensitive information, or high-risk applications.Source: ETHICS STATEMENT · p. 11
• Real-world experiments were conducted in a controlled laboratory under standard robot-safety procedures, and the authors state they followed the ICLR Code of Ethics.Source: ETHICS STATEMENT · p. 11

• The supplementary material describes the masked-action formulation and inference procedure.Source: REPRODUCIBILITY STATEMENT · p. 11
• It documents policy architectures, training settings, mask construction, evaluation metrics, and the real-world setup.Source: REPRODUCIBILITY STATEMENT · p. 11
• Ablation descriptions distinguish matched-setting comparisons from historical comparisons with differing conditions and identify results aggregated across configurations and checkpoints.Source: REPRODUCIBILITY STATEMENT · p. 11

1 ABLATION STUDIES

• The ablations vary the MAP encoder, forward window length, progress module, training schedule, and progress source; some settings differ in multiple factors.Top-k SR averages the k highest evaluated SR values within a run, while lower CE indicates lower control error.Source: §1 · p. 15

1.1 MAP WINDOW ENCODING AND FEATURE DIMENSION

• 19.00%, 14.67%, and 22.00% are the 2DConv Top-3 SRs at dimensions 32, 64, and 128; at dimension 32, 2DConv exceeds 1DConv’s 12.00%.The 128-dimensional encoder was evaluated later, at 275k updates, so encoder and training configuration both differ.Source: §1.1 · p. 15, Table 2 · p. 15

1.2 MAP WINDOW LENGTH

• 52 locations yield a 70.22% Top-3 SR, exceeding 64- and 70-location windows by 8.89 and 10.00 percentage points, respectively.At the final checkpoint, the 52-location window also has the highest SR (63.33%) and end SR (36.33%), and lowest CE(all) (0.02317).Source: Table 3 · p. 15, §1.2 · p. 16

1.3 PROGRESS-MODULE CONFIGURATION

• 74.00% mean SR for STPM v3 leads v5 at 70.67% and v6 at 69.33% with a fixed IP checkpoint.Architecture, observation history, and checkpoint selection vary together, so the comparison cannot isolate architecture’s contribution.Source: §1.3 · p. 16, Table 4 · p. 16

1.4 JOINT AND TWO-STAGE TRAINING

• 56.00%/54.40% Top-3/Top-5 SR with two-stage training compares with 50.00%/47.60% for joint training on 500 demonstrations.A separate 1,000-demonstration run reached a best evaluated SR of 100.00% at 150k; dataset size was not the only factor that differed between runs.Source: §1.4 · p. 16, Table 5 · p. 16

1.5 ESTIMATED PROGRESS VERSUS ROLLOUT TIMESTEP

• 16 percentage points: STPM alignment raises success-once SR from 67% to 83% and reduces CE(all) from 0.03983 to 0.02912.For this checkpoint and evaluation set, learned progress alignment improves task success and overall reference accuracy.Source: §1.5 · p. 17, Table 6 · p. 17

2.1 BENCHMARKS, DATA, AND BASELINES

• ManiSkill covers object manipulation and LIBERO-10 contains ten long-horizon tasks; training and test references are created by masking demonstrations, while STC provides a matched-backbone direct-conditioning baseline.Within each comparison, methods use the same data, random seed, and initial states.Source: §2.1 · p. 17

2.2 POLICY ARCHITECTURES AND TRAINING

• ManiSkill uses separate policies trained on 500 demonstrations, while LIBERO-10 uses a shared language-conditioned policy with random-mask training followed by structured-mask mixtures.The progress module estimates task progress from frozen CLIP image and text features and proprioceptive states.Source: §2.2 · p. 18

2.3 CONTROL-ERROR DEFINITIONS

• Control error measures each reference signal's closest-trajectory-point mismatch rather than prescribed-timestep alignment; real-world task errors instead report physical distance or tilt angle.Lower CE means closer reference agreement, and CE is not reported for unconditioned DP and DiT baselines.Source: §2.3 · p. 18, §2.6 · pp. 20–21

2.4 MASK CONSTRUCTION AND INFORMATION BUDGETS

• The spatiotemporal information budget counts visible scalar action entries, so equal visibility ratios can encode unequal information for structured and random masks.The budget-matched comparison uses NST = 1.4T; the unseen Mix0 evaluation combines dense 2D flow, one 7D pose, and three 3D points without fine-tuning.Source: §2.4 · p. 19

2.5 ADDITIONAL JOINT-TRAINING RESULTS

• Inpainting Policy has the highest ManiSkill success rate on LIFTPEGUPRIGHT and PICKCUBE, while DiT Policy performs best on POKE-CUBE.The per-task comparison uses separate policies trained with 500 demonstrations per task.Source: §2.5 · p. 19, Table 8 · p. 20
• IP-RePaint reduces control error by 42.2% on LIBERO-10 and 39.1% on ManiSkill, with lower success rates in both settings.For mask-type results, Inpainting Policy averages 78.67% SR and reaches 80.56% on unseen Mix0; IP-RePaint reaches 52.64% on dense 2D/3D flow and 64.17% on Mix0.Source: §2.5 · pp. 19–20

2.6 REAL-WORLD SETUP AND EVALUATION

• The real-world system maps user references and progress values into MAP windows that condition IP alongside observations and robot state; requests should remain compatible with the training distribution and with one another.The experiments include cup arrangement and controlled pouring, with physical reference errors reported separately from normalized action-space CE.Source: §2.6 · pp. 20, 22
• In cup arrangement, IP exceeds 93% success with three or more reference points, while single-point final-stage guidance yields below 30%; pouring reaches 93.33% with four orientation references and 100% with eight or sixteen.Cup-to-reference-point distance averages 44.3 mm; pouring angle errors range from 7.6867° to 7.8133°.Source: §2.6 · p. 21

3.1 PROGRESS-GUIDED MAP ALIGNMENT

• Estimated task progress selects a local MAP window, aligning references with task stages despite variation in execution speed.The encoder receives masked values and visibility masks, preserving the distinction between unspecified entries and specified zeros.Source: §3.1 · p. 23
• The local reference window includes past and future context around the progress-selected anchor, with its coverage controlled by the window lengths.Indices are clipped to the reference range, and boundary actions and masks are repeated to preserve fixed window length.Source: §3.1 · p. 23

3.2 REPAINT INFERENCE

• RePaint replaces specified entries with appropriately noised reference values at every reverse diffusion step and uses clean values in the final sampled action chunk.This guarantees the specified sampled action values, not exact satisfaction during physical execution.Source: §3.2 · p. 23

3.3 MASKED DENOISING OBJECTIVE

• The masked denoising objective trains predictions on both visible and missing action regions, encouraging reference consistency and reconstruction of unspecified actions.The visible-token objective is learned, unlike RePaint's explicit overwriting of visible values during sampling.Source: §3.3 · p. 24

3.4 MASK FAMILIES FOR JOINT TRAINING

• Joint training samples structured, no-reference, and random-token masks, with random masks exposing the policy to partial specifications beyond predefined control modalities.The no-reference family trains operation without external action references; LIBERO-10 uses random masks first, then a mixture of structured masks.Source: §3.4 · pp. 24–25

3.5 GENERALIZATION TO UNSEEN MASK COMPOSITIONS

• Because compatible references share one action representation, their masks can be combined by union, allowing evaluation of compositions excluded from training without fine-tuning.Overlapping specified entries are assumed to agree.Source: §3.5 · p. 25

3.6 CONDITIONING ARCHITECTURE

• The denoising condition concatenates visual, proprioceptive, and MAP features; the MAP encoder produces a 128-dimensional vector from the local masked-action window.The formulation is instantiated with either a Diffusion Transformer or a U-Net-based Diffusion Policy.Source: §3.6 · p. 25