Mind the Execution Gap: Action-Semantic Mismatch in World-Model Control

1 INTRODUCTION

• World-model controllers use action-conditioned predictions and planning, while practical control systems may execute actions on different timelines from those on which commands are issued.This leaves a gap between the action supplied to the model and the action that generates the transition.Source: §1 · p. 1
• The paper frames an execution-semantic gap: planning-based models consume future action sequences, whereas recurrent models condition state updates on previous actions, so the mismatch may differ by architecture.The introduction names the resulting modes a future-action timeline mismatch and a previous-action attribution mismatch.Source: §1 · p. 2
• Execution-Consistent Interfaces use Future-Sequence Execution for planning-based models and Applied-Action Feedback for recurrent models, changing the controller–execution interface without modifying pretrained world models.Future-Sequence transmits the planner’s future action suffix; Applied-Action Feedback supplies the applied action to the recurrent update.Source: §1 · pp. 2–3
• Experiments report that the corrections address distinct architecture-specific mismatches across TD-MPC/TD-MPC2, multiple control domains, measured network traces, and a process-separated asynchronous stack.The reported conditions also include delay, packet loss, and reordering.Source: §1 · pp. 2–3

2 WHY DO WORLD-MODEL CONTROLLERS FAIL UNDER ASYNCHRONOUS EXECUTION?

• The analyses frame asynchronous execution as changing the action information consumed by the world model, whose applied action depends on command history and execution state.Planning models roll future actions through latent dynamics; recurrent models condition state updates on the previous action and new observation.Source: §2 · p. 3

2.1 FINDING 1: PLANNING-BASED WORLD MODELS NEED THE CORRECT FUTURE ACTION TIMELINE

• Increasing action-delivery delay raises command–applied divergence and lowers short-horizon return even when TD-MPC2 replays the same planned command sequence from matched states.Figure 2 varies delay while holding the command sequence fixed, isolating execution from planner behavior.Source: Figure 2 · p. 3, §2.1 · p. 3
• For TD-MPC2, current applied-action or buffer information is insufficient; the exact future applied-action sequence improves control only when used in latent dynamics rollout, not trajectory rescoring.This identifies a mismatch between the action sequence used to predict future states and the sequence asynchronous execution realizes.Source: Figure 3 · p. 4, §2.1 · p. 4

2.2 FINDING 2: EXECUTION-SEMANTIC MISMATCH MANIFESTS DIFFERENTLY ACROSS WORLD-MODEL ARCHITECTURES

• For DreamerV3, applied-action feedback improves prediction and return for applied-action-trained checkpoints but is not uniformly beneficial for command-trained models.The result links the recurrent update’s action input to the action semantics learned during training.Source: Figure 4 · p. 4, §2.2 · p. 4

3.1 FUTURE-SEQUENCE EXECUTION FOR PLANNING-BASED WORLD MODELS

• TD-MPC2 packets carry the current action, remaining future actions, and a generation index so execution can continue the latest plan through delay, loss, or reordering.The actuator selects from the newest valid arrived sequence; a newer sequence replaces the unexecuted remainder, and stale arrivals are discarded.Source: §3.1 · p. 5
• Future-sequence execution preserves the planner’s already-produced future control information rather than predicting communication delays or adding a control policy.When a replanned command is unavailable, the actuator continues the newest available plan, reducing the mismatch between rollout and execution timelines.Source: §3.1 · p. 5
• Table 1 reports TD-MPC2 returns averaged across tasks and normalized to matched clean SINGLE-ACTION returns across clean and adverse execution conditions.Its listed conditions are random delay, bursty delay, packet loss, reordering, and adverse average.Source: Table 1 · p. 6, §3.1 · p. 6

3.2 APPLIED-ACTION FEEDBACK FOR RECURRENT WORLD MODELS

• DreamerV3’s RSSM can pair an observed transition with the wrong action when asynchronous execution applies an action other than the issued command.Applied-Action Feedback substitutes the action actually applied for the command action in the recurrent update.Source: §3.2 · p. 6
• An execution receipt accompanies the subsequent observation and identifies the action that generated the transition by packet source and position.It can carry the action vector or an identifier for recovering it from local command history, without an additional communication round trip.Source: §3.2 · p. 6
• When the receipt arrives in time, the RSSM updates with the action that generated the transition, while the pretrained world model and actor remain unchanged.The interface exchanges information between the controller and execution system; it does not provide future execution information to DreamerV3.Source: §3.2 · p. 6

4.1 SETUP

• The setup freezes pretrained controllers except when studying delay-aware retraining, and evaluates delays, packet loss, reordering, measured traces, and a process-separated asynchronous stack.Primary experiments use the DeepMind Control Suite, with additional cross-domain experiments and episodic return, task success, and command–applied divergence metrics.Source: §4.1 · pp. 6–7

4.2 EXPERIMENTAL RESULTS

• Future-Sequence improves TD-MPC2 across adverse execution conditions, while Applied-Action Feedback gives DreamerV3 its clearest gains under random delay, packet loss, and reordering.For TD-MPC2, it has the highest adverse-condition average among reported interfaces excluding the Oracle; for DreamerV3, fixed-delay improvements are less uniform.Source: §4.2 · p. 7
• Future-Sequence also gains on TD-MPC, held-out MetaWorld, ManiSkill2, and MyoSuite tasks, and with measured network-delay profiles.These results extend beyond TD-MPC2, a single task domain, and synthetic delay.Source: §4.2 · p. 8
• In a process-separated UDP stack, Future-Sequence recovers TD-MPC2 performance across evaluated perturbations, while Applied-Action Feedback improves DreamerV3 in most conditions.Actuator-originated ECT-FULL and ECT-ID receipts recover the exact applied action across 30,000 tested transitions.Source: §4.2 · p. 8, Figure 7 · p. 8

4.3 ANALYSIS

• Future-Sequence benefits clean-trained checkpoints broadly, but combining it with Fixed-2 training degrades every adverse condition; random-delay training leaves the adverse average nearly unchanged.The results indicate that delay-aware training and execution-side correction are not automatically additive.Source: §4.3 · p. 8
• Immediate applied-action feedback gives the highest aggregate return at both evaluated control rates, and one- or two-period delays progressively reduce performance.The analysis identifies timely, transition-aligned feedback—not access to applied actions alone—as important for recurrent-state correction.Source: §4.3 · p. 8, Figure 8 · p. 8
• Future-Sequence and Receding Buffer match at reported precision under random delay, bursty delay, and reordering, while Future-Sequence performs better under packet loss.Both outperform transmitting isolated actions or executing fixed chunks; their buffer-management differences further shape robustness.Source: §4.3 · pp. 8–9

5.1 DELAYED REINFORCEMENT LEARNING.

• Delayed reinforcement-learning methods augment state, predict delayed effects, or adapt to stochastic and partially observable delays, with effective decision making under delay as their primary objective.The literature progresses from representing delayed interactions to inferring decision-relevant states and actions under uncertain execution timing.Source: §5.1 · p. 9

5.2 WORLD-MODEL CONTROL UNDER ASYNCHRONOUS EXECUTION.

• Prior world-model and robot-learning methods mainly compensate for delay or align prediction and execution; this paper studies how command–applied divergence changes action semantics in planning-based and recurrent models.The identified mismatches are future-action timeline mismatch in planning-based models and previous-action attribution mismatch in recurrent models.Source: §5.2 · pp. 9–10

6 CONCLUSION

• Asynchronous execution disrupts action-conditioned computation in world-model controllers.The paper frames this as an action-semantics issue within the controller.Source: §6 · p. 10
• Controlled interventions identify a future-action timeline mismatch in TD-MPC2 and a training-dependent previous-action attribution mismatch in DreamerV3.The two architectures therefore exhibit distinct execution-related mismatches.Source: §6 · p. 10
• Execution-consistent interfaces improve control across tasks and asynchronous execution conditions, highlighting alignment of execution-time action information with model computation and learned action semantics.Source: §6 · p. 10

A DREAMERV3 NUMERICAL RESULTS

• The normalization divides each task's return difference by its mean clean-execution return under Command Feedback.Source: A DREAMERV3 NUMERICAL RESULTS · p. 12
• The reported task average weights Walker, Cheetah, and Cup equally.Source: A DREAMERV3 NUMERICAL RESULTS · p. 12
• Table 3 reports DreamerV3 clean-normalized return improvement over Command Feedback, with equal task weights and ten evaluation seeds per task.Source: Table 3 · p. 12

B DELAY-AWARE TRAINING NUMERICAL RESULTS

• Table 4 provides the absolute returns underlying Table 2.Source: B DELAY-AWARE TRAINING NUMERICAL RESULTS · p. 12
• Table 4 reports Walker episodic returns across training and execution conditions, averaging five independently trained checkpoints and ten evaluation seeds per checkpoint.F1 to F3 denote fixed delays, and Adv. is the mean over all non-clean conditions.Source: Table 4 · p. 12

C IMPLEMENTATION DETAILS

• Execution-interface comparisons keep controller parameters fixed, pair methods by environment and transport seeds, and reset queue, held action, and controller state at episode starts.Controller training and interface evaluation are separate experiments.Source: C IMPLEMENTATION DETAILS · p. 13
• TD-MPC2 diagnostics use a five-step horizon, 64 candidate trajectories, eight policy-prior trajectories, and three base planning iterations.For action dimensions of at least 20, two additional iterations are used; returns use 500-step episodes, with the planner random state preserved during diagnostics.Source: C IMPLEMENTATION DETAILS · p. 13
• DreamerV3 aligns the recurrent update’s action with the transition producing the current observation, distinguishing the issued command from the action actually applied.The aligned action is distinct from the next command generated after observing the transition.Source: C IMPLEMENTATION DETAILS · p. 13

D BASELINE DEFINITIONS

• FUTURE-SEQUENCE stores actions in intended execution slots, updates overlapping future slots with newer plans, and rejects stale plans.SINGLE-ACTION sends only the first planned action; HOLD-LAST reuses the latest available action, FIXED CHUNK waits for queue exhaustion, and RECEDING BUFFER replaces the remaining queue with newer plans.Source: D BASELINE DEFINITIONS · p. 13
• The FUTURE-ACTION ORACLE evaluates candidates using an isolated network and actuator copy, then supplies the resulting applied-action sequence to world-model rollout.It is a privileged diagnostic, not a deployable interface or guaranteed upper bound on return.Source: D BASELINE DEFINITIONS · p. 13
• DreamerV3 CMD supplies the issued command to the recurrent update, while APP supplies the actual previous action; RCP sends five copies of the current command.In the process-separated implementation, ECT-FULL returns the applied-action vector, whereas ECT-ID returns its source packet and position for reconstruction from command history.Source: D BASELINE DEFINITIONS · p. 13

D.1 EXTERNAL LEARNING-BASED BASELINES

• DATS uses five probabilistic dynamics models and CEM planning, prepending pending actions to candidate suffixes to account for committed execution.The models have four hidden layers of width 200; planning uses horizon 30, 500 candidates, 50 elites, and five optimization iterations, with training under a fixed two-step delay.Source: §D.1 · p. 13
• ACDA uses a delay-indexed action matrix and produces its native delay-conditioned matrix rather than a TD-MPC2 future sequence.Its configuration includes a horizon of four, policy hidden layers of width 256, predictive-model hidden and recurrent dimensions of 256 and 384, and a model window of 16.Source: §D.1 · pp. 13–14
• DATS and ACDA train their own predictive models and controllers, whereas Future-Sequence uses pretrained TD-MPC2 without parameter changes.Each method is evaluated as a complete controller–execution system; checkpoints remain fixed, with ten environment seeds per checkpoint and condition and 500 control decisions per episode.Source: §D.1 · p. 14

• The fixed-command intervention replays one planned command sequence from matched states while varying delivery delay from zero to three steps.It measures command–applied divergence and the ten-step return change relative to zero-delay execution.Source: E ABLATION PROTOCOLS · p. 14
• The execution-information ablation compares current applied action, actuator-buffer actions, and an exact future sequence mapped through a copied execution system.The actual execution mechanism is held fixed across information conditions, with 20 paired episodes per perturbation.Source: E ABLATION PROTOCOLS · p. 14
• The insertion-point ablation contrasts mapped actions used in reward scoring only, latent dynamics rollout only, or both components.All variants share the terminal policy and value-estimation rule; FULL INTEGRATION changes where execution information enters planning without additional model training.Source: E ABLATION PROTOCOLS · p. 14
• The prediction diagnostic fixes the observation trajectory and forecast action while changing only the recurrent update’s previous-action input.Walker and Cheetah use command-trained and applied-action-trained checkpoints, with one held-out episode and 950 prediction steps per checkpoint–task condition.Source: E ABLATION PROTOCOLS · p. 14
• Control improvement is evaluated separately over ten paired closed-loop episodes, with 95% confidence intervals estimated from 20,000 paired bootstrap resamples.Prediction bars report descriptive means without confidence intervals; M denotes observation-prediction MSE.Source: E ABLATION PROTOCOLS · p. 14
• The training–interface comparison evaluates five checkpoints for each of three Walker training conditions under both execution interfaces.Each checkpoint-condition uses ten seeds, and the adverse average equally weights six non-clean evaluation conditions.Source: E ABLATION PROTOCOLS · p. 14
• The process-separated stack tests 10 and 20 Hz operation on Walker, Cheetah, and Cup under fixed and random delay, packet loss, and reordering.Conditions include 20% packet loss with random delay and selected packets delayed by two periods with probability 0.60; feedback latency compares immediate, one-period, and two-period-delayed action feedback.Source: E ABLATION PROTOCOLS · pp. 14–15
• Main DreamerV3 returns are normalized by each task’s mean clean CMD return and averaged equally across tasks.Process-separated recovery is computed separately; return uncertainty uses complete episodes as sampling units rather than individual transitions.Source: E ABLATION PROTOCOLS · p. 15