Decoupling Planning and Control for Instructable Agents

Zineng Tang; Kelsey R. Allen; Sjoerd van Steenkiste; Ishita Dasgupta; Alane Suhr

Source-linked AI summary

Decoupling Planning and Control for Instructable Agents

Zineng Tang, Kelsey R. Allen, Sjoerd van Steenkiste, Ishita Dasgupta, Alane Suhr

arXiv:2608.26788v1Aug 27, 2026cs.AIcs.CLcs.MAcs.RO

TL;DR

VLMs provide high-level planning but struggle with reliable low-latency action, while world-model controllers lack open-ended guidance. Instruct-to-Act combines them by training an instruction-conditioned controller with synthetic rollout annotations and existing world-model objectives, outperforming direct VLM action generation while preserving fast control and modular planner swapping.

Problem
Embodied agents need both high-level task reasoning and sustained high-frequency control, but pretrained VLMs struggle with environment-specific action streams.
Method
Instruct-to-Act composes a pretrained VLM planner with an environment-specific instruction-conditioned RSSM controller trained using reward, world-model, and behavior-cloning objectives.
Results
Planner–controller agents significantly outperform direct VLM action generation in task score and inference efficiency and achieve competitive performance in 6 of 7 environments.
Takeaways & Limitations
Decoupling planning and control supports strong task performance, low latency, modular planner evaluation, and language-based multi-agent coordination.
Takeaways & Limitations
A jointly trained controller across all environments lost 1.12 points on Minecraft Diamond, about 10% relative, compared with environment-specific training.

Abstract

from arXiv · show

Recent work shows that pre-trained, instruction-tuned vision-language models (VLMs) perform well at mapping from instructions and observations to high-level plans, but struggle to realize such plans as reliable low-latency action sequences in unfamiliar environments. At the same time, world-model controllers excel at fast observation-to-action control, but lack open-ended task guidance. In this work, we combine these strengths into a single system, Instruct-to-Act, where we train a world-model controller to act autonomously at high frequency when conditioned on sparse, higher-latency, and high-level text instructions generated by a VLM planner. To train controllers to be language-instructable, we relabel segments of controller policy rollouts with synthetic instructions and jointly optimize a behavior-cloning objective along with existing reward-maximizing and world-modeling objectives. We evaluate our proposed approach across seven embodied environments, including three multi-agent environments where VLM planners coordinate through language while trained controllers serve as their actuators. Under matched observation and action spaces, our decoupled approach consistently outperforms controller-only and direct VLM action-generation variants, preserves fast control, and lets us swap in different pretrained VLM planners without fine-tuning, while remaining competitive with strong vision-language-action and multi-agent RL baselines on six of seven tasks.

1 Introduction

Instruct-to-Act decouples high-level VLM planning from real-time world-model control through a language-conditioned controller. This supports modular planner–controller inference, including real-time multi-agent coordination.

Instruct-to-Act separates VLM-generated plans and instructions from controller-generated low-level actions.The planner operates at higher latency while the controller processes observations and acts in real time.
The controller is trained as a language-aware world model and policy conditioned on planner instructions.Representation learning, actor control, and critic/value prediction use a latent instruction representation.
Decoupled inference lets multiple language planners share controller architectures for real-time communication and coordination.The design supports evaluating VLMs as both individual embodied agents and coordinating multi-agent planners.
92.8% average instruction-following accuracy was achieved across instructions generated by diverse VLM planners.Controllers were first evaluated independently of planner action generation, then paired with VLM planners across embodied tasks.
Planner–controller agents outperformed direct high-level-to-low-level VLM action generation and were competitive in 6 of 7 environments.The reported benefits include task score, inference efficiency, low latency, and modular evaluation across heterogeneous environments.
Only lightweight environment-specific controllers are trained, while VLM planners remain untuned to low-level action spaces.The language interface supports asynchronous planner swaps and direct multi-agent coordination through a shared controller architecture.

2 Overview and Experimental Setup

The paper targets embodied tasks requiring both high-level decisions and sustained, high-frequency control. It composes an instruction-producing VLM planner with an environment-specific recurrent controller and evaluates the system across seven environments.

At inference, the planner maps observations to natural-language instructions, and the controller maps instructions and visual observations to low-level actions.This composition avoids requiring VLMs to generate environment-specific low-level actions directly.
The central challenge is sustaining long low-level action streams at control rate while selecting appropriate subgoals.Pretrained VLMs struggle when action spaces are high-frequency and environment-specific.
The policy samples an instruction from πp and then samples low-level actions from πc conditioned on observations and that instruction.An instruction-completion action triggers replanning; without an instruction, the controller can continue autonomously.
The controller is an instruction-conditioned RSSM trained with post-hoc VLM-labeled rollout segments and a behavior-cloning objective.This objective is combined with Dreamer world-model, actor, and value losses, without expert demonstrations.
Evaluation spans seven environments, including classic single-agent benchmarks and cooperative video-game-based multi-agent tasks.

3 Method

The method runs planning and control as separate processes, with VLMs issuing language instructions asynchronously to an instruction-conditioned RSSM controller. Controllers learn from reward, world modeling, and behavior cloning on synthetic instruction annotations.

3.1 Decoupling Planning and Control in Inference: Planner and controller inference run in independent parallel processes, allowing low-level actions while planners reason and communicate.The same architecture extends to multi-agent settings by spawning planner and controller threads per agent.
3.1 Decoupling Planning and Control in Inference: Asynchronous planners use live observations and emit the next instruction when the controller completes its current one.This avoids waiting for controller completion before beginning the next planning computation.
3.1 Decoupling Planning and Control in Inference: Multi-agent planners coordinate through language while controllers share parameters but maintain separate observations, latent states, and actions.The setup does not use a centralized critic or structured role labels.
3.2 Controller Architecture: The RSSM optionally conditions policy and value heads on embeddings from a frozen pretrained contrastive language encoder.A learned null embedding allows autonomous controller behavior when no instruction is available.
3.2 Controller Architecture: The controller output includes an instruction-completion indicator used to determine when a new instruction should be issued.A small head predicts completion from the latent state.
3.4 Training: Training alternates on-policy rollout collection with optimization of reward, world-model, and instruction-following objectives.Behavior cloning uses VLM-generated instructions assigned post hoc to variable-length rollout intervals.
3.4 Training: Half of the replay buffer is annotated, while the remainder trains autonomous behavior from environment reward.Annotation adds roughly 17% GPU-hours and slows training by 12%.

4 Experiments

The experiments evaluate language-conditioned controllers and VLM planners across embodied tasks, scaling settings, instruction sources, and cross-planner transfer. Results show consistent benefits from planning-control decomposition, efficient online planning, robust transfer, and high instruction-following reliability, while per-environment controllers remain necessary.

Architecture and Training: One 800M-parameter controller is trained per environment, using post-hoc GPT-4o annotations on 50% of replay segments.Rollouts use empty language during collection; annotated segments have lengths from 1 to 20.
VLMs as Planners: Decomposition improves over direct VLM action generation, while language guidance improves over controller-only baselines across tasks.GPT-4o sparse instructions executed by a trained controller improve performance on every task.
Scalability and Efficiency: Decentralized communication improves Pico Park task performance while maintaining throughput equivalent to centralized control as agents scale from 2 to 8.Each agent pairs a Qwen-VL-2.5-72B planner with an 800M-parameter controller.
Scalability and Efficiency: Online planning improves Minecraft Diamond performance over offline planning while remaining an order of magnitude faster in throughput across model sizes.The comparison uses small, medium, and large Qwen-VL-2.5 planner-controller configurations.
Generalization: Instruction-following accuracy remains between 172/200 and 194/200, or 86% to 97%, across domains and planners.Planner-decided instruction timing also outperforms fixed cadence on Minecraft Diamond, including 10.75 versus 9.60 for GPT-4o.
Generalization: Environment-derived symbolic labels slightly outperform VLM-generated instructions, but VLM annotation requires only raw observations and supports domains without environment metadata.Random strings and no instructions significantly underperform semantic instruction sources.
Generalization: Cross-planner transfer remains stable across all planner pairs, with scores ranging from 9.2 to 10.2.This indicates the controller does not overfit to the annotating planner’s phrasing.
Generalization: A single controller trained jointly across all environments loses 1.12 points on Minecraft Diamond, motivating lightweight domain-general controllers as future work.The reported per-environment training cost is approximately 23 GPU-hours on four RTX A6000 GPUs.

5 Related Work

Related work commonly separates slow reasoning from fast control, while language-conditioned and open-ended agents connect planning with embodied actions. Instruct-to-Act distinguishes itself through a natural-language, plug-and-play interface, a world-model controller, and post-hoc relabeling without expert data.

Slow/Fast Decomposition: Slow reasoning and fast reactive control are established across autonomous driving, robot manipulation, and vision-language navigation.HiRT is cited as an asynchronous planner-controller system communicating through latent representations.
Planner–Controller Interfaces: Instruct-to-Act uses natural language rather than latents or waypoints, enabling planner interchange and language-based multi-agent coordination.The paper explicitly does not claim slow/fast decomposition itself as novel.
Controller Design: Its fast component is a world-model controller intended to generalize across heterogeneous domains rather than a domain-specific stack or waypoint follower.The controller is trained without expert data, using post-hoc VLM relabeling of its own on-policy rollouts.
Language-Conditioned Control: Prior language-conditioned systems compose language-model reasoning with low-level skills, whereas end-to-end VLA systems directly map observation-language inputs to actions.The related systems include SayCan, Code as Policies, Inner Monologue, RT-1, RT-2, RT-H, OpenVLA, π0, GR00T, and Octo.
Open-Ended Agents: Open-ended planner–controller agents are often tightly coupled to particular domains, planner backbones, or action-generation pipelines.The paper instead emphasizes a reusable language interface with asynchronous planning and controller execution.
Alternative Interfaces: Language provides an alternative to latent goal or skill abstractions by allowing humans and off-the-shelf VLMs to plan without retraining the controller-side interface.This property supports the paper’s plug-and-play evaluation setup.

6 Conclusion

Instruct-to-Act combines low-latency world-model control with high-level VLM reasoning through language conditioning. Its RSSM-based controller learns from imagined rollouts and post-hoc instruction annotations, while supporting autonomous action when instructions are absent.

6 Conclusion: Instruct-to-Act combines world-model controllers’ low latency with VLMs’ high-level reasoning and extends the interface to multi-agent coordination.The framework is designed to translate abstract guidance into long low-level action sequences without specialized MARL training.
World-model control: The controller maintains a latent state from deterministic and stochastic components, using posterior states during training and prior states during imagination.The world model decodes observations, rewards, continuations, and optional next instructions from latent states.
Learning objectives: Training combines reconstruction and prediction terms with KL regularization, then optimizes Dreamer-style actor and value objectives on imagined trajectories.The policy and value functions are conditioned on language, and gradients propagate through imagined dynamics.
Implementation: The implementation uses a frozen MiniLM language encoder and DreamerV3-style controller configurations ranging from 50M to 800M parameters.The language embedding conditions both the world model and control pathway, with optimizer and update schedules following the stated hyperparameters.

A.4 Training Compute and Annotation Overhead

Training uses post-hoc instruction annotation while keeping compute overhead modest relative to controller-only training.

A.4 Training Compute and Annotation Overhead: Post-hoc annotation adds roughly 17% GPU hours and causes a 12% training slowdown relative to controller-only training.Across tasks, one environment requires approximately 36 wall-clock hours at most and 23 hours on average.

A.5 Additional Results

Additional experiments show that the framework preserves throughput, supports multiple instruction regimes and controller architectures, and maintains strong instruction-following reliability.

Efficiency: Instruct-to-Act achieves the best throughput among representative end-to-end or tightly coupled baselines.The controller executes every environment step while the planner is queried only sparsely.
Instruction modes: Instruction modes compare fixed cadence, VLM-decided proactiveness, and fully VLM-controlled timing on Minecraft Diamond.The VLM-decided mode emits instructions based on uncertainty and enforces a minimum gap between instructions.
Controller architecture: A transformer world-model controller yields similar trends, whereas removing the world-model structure degrades performance.This indicates that the broader framework is not tied specifically to Dreamer or RSSM controllers.
Instruction following: Instruction-following accuracy remains around 86–97% across domains and planners in human evaluation.The most common failures involve vague or underspecified instructions that produce locally suboptimal behavior until clarification arrives.

A.6 Scalability and Efficiency Curves

Scaling and efficiency studies find a favorable performance-throughput frontier, with online planning improving throughput despite instruction staleness. Multi-agent failures are dominated by planner-level coordination errors rather than controller execution failures.

Scalability and efficiency curves: Configurations that improve task performance usually retain most efficiency gains from decoupled control rather than adopting end-to-end VLM latency.On Minecraft Diamond, performance improves monotonically as either planner or controller size grows under matched settings.
Asynchronous planning: Online planning achieves 2.8× higher throughput on Minecraft Diamond and 1.8× on Overcooked than offline planning.Instruction staleness is roughly 51 steps for Minecraft and 165 steps for Overcooked, while the controller continues executing the previous instruction.
Multi-agent failure modes: Planner-level coordination failure dominates Overcooked failures, with 96 of 100 windows showing near-duplicate sub-task assignments.Only 3 of 100 windows show controller instruction-following failures.
Instruction characteristics: Minecraft instructions are dominated by local action descriptions, which account for 82.8% of 47,547 instruction instances.The remaining 17.2% are high-level subgoals chaining a primary action to a downstream task.
Instruction-length generalization: The stop head remains robust for instruction lengths from L=2 through L=50, with degradation beginning gradually above L=50.The evaluation used 100 instructions across 25 length checkpoints and averaged results over 5 seeds.

A.10 Training Dynamics of the Composite Loss

The composite objective remains stable while its dominant loss component shifts from world-model reward prediction to behavior cloning on instruction-relabeled segments. The appendix also identifies the prompts used for training annotation, online inference, and multi-agent communication.

Training dynamics: Early training is dominated by the world-model reward-prediction loss, while behavior-cloning becomes dominant as instruction-relabeled segments accumulate.The composite objective combines world-model, actor, value, behavior-cloning, and stoptoken losses.
Training dynamics: The dynamics loss initially spikes and then steadily decreases as the policy learns environment dynamics after early exploration.The authors attribute the spike to unexpected observations encountered during exploration.
Training dynamics: Policy and behavior-cloning objectives continue improving without any loss component collapsing or diverging, indicating stable training under fixed loss weighting.No additional loss re-weighting or scheduling was used in this run.
Prompt usage: The summarizer, inference, and multi-agent prompts support post-hoc annotation, online instruction issuing, and multi-agent communication, respectively.The multi-agent prompt is appended to the single-agent inference prompt.

B Tasks Details

The evaluation spans seven embodied environments covering single-agent control, navigation, survival, and cooperative multi-agent tasks. Each environment specifies its own goals, observation-action interface, protocol, and metric, while the planner-controller system supports asynchronous execution and communication.

Benchmark scope: The benchmark covers seven environments, with each described by its goal, observation space, action space, evaluation protocol, and metric.The suite includes both single-agent and multi-agent settings.
Atari 2600: Atari evaluates episode game scores from raw visual frames using the standard ALE protocol and primitive action set.Evaluation uses fixed episodes and capped frames, with ALE-specific procedures when applicable.
Minecraft Diamond: Minecraft Diamond requires obtaining a diamond from scratch using first-person RGB, inventory observations, and combined camera, navigation, mining, crafting, and smelting actions.Episodes end on death, diamond acquisition, or an 18,000-frame limit; performance averages over 500 unseen-seed episodes.
Crafter: Crafter measures broad survival-world progress through the geometric mean of 22 achievement success rates under reward-based and reward-free tracks.Agents use local top-down pixels, inventory status, and discrete movement, interaction, crafting, sleeping, and placing actions.
DM-Lab: DM-Lab tests first-person 3D navigation to goals under partial observability, commonly using held-out maps and return, success-rate, and path-efficiency metrics.The action interface includes continuous look and movement controls.
Overcooked: Overcooked evaluates two-player cooperative cooking, including zero-shot coordination with unseen partners and layouts using team return, success, and cross-play ZSC metrics.The task uses gridworld or egocentric observations and discrete move, interact, and pick/place actions.
Puzzle-platforming: Cooperative puzzle-platforming requires synchronized actions among 2–8 players, with level completion rate and median completion time as primary metrics.Evaluation spans fixed multiplayer levels, multiple seeds, and player configurations, requiring every agent to reach the goal.
Multi-agent Minecraft: Multi-agent Minecraft collaboration covers language-enabled Cooking, Crafting, and Construction tasks with split resources or knowledge, scored by category success and blueprint alignment.Procedural tasks vary recipe complexity, information asymmetry, and construction materials or skills.

D Instruction-following evaluation protocol.

Instruction-following accuracy is assessed by human judgment over held-out planner-generated instructions, with model agreement used only as a sanity check. Additional details specify segment sampling and communication limits.

D Instruction-following evaluation protocol.: Instruction-following accuracy is judged on 200 held-out planner-generated instructions per domain.The evaluator reviews the instruction, observation or trajectory segment, and real-time controller video.
D Instruction-following evaluation protocol.: The evaluator determines whether the controller’s behavior completed the intended instruction.The judgment uses both the instruction and its corresponding execution evidence.
D Instruction-following evaluation protocol.: GPT-5.5 agreed with the human annotator on 93.2% of cases across tasks, providing a model-based sanity check.This agreement does not replace independent multi-annotator human evaluation.
D Instruction-following evaluation protocol.: The authors explicitly qualify the accuracies because evaluation combines one annotator with model agreement rather than independent multi-annotator assessment.This is presented as a sanity check, not a substitute for broader human evaluation.
D.1 Additional Training and Communication Details: Training rollouts may associate different timesteps with different instruction embeddings and completion indicators, while only the final timestep is marked done.Annotation segments use randomized lengths, uniformly sampled starts, and non-overlap constraints.
D.1 Additional Training and Communication Details: Each decentralized planner may initiate at most one message to another planner per VLM inference step.In centralized communication, a fixed hub sends messages to all other agents.