Source-linked AI summary
TaskFrame: Connecting Task-Level Planning with Frame-Level Control for Game Agents
Guangzhi Wang, Junfu Pu, Yiran Wang, Xingyilang Yin, Honghao Chen, Cheng Cheng, Lingen Li, Yinan Zhou, Jiarui Wu, Yinuo Jiang, Hongbo Jin, Xiaodong Cun, Ying Shan
TL;DR
TaskFrame addresses the challenge of combining high-level task reasoning with low-latency control in changing visual environments. It connects an instruction-decomposing planner to a frame-level controller through cached goal embeddings. In Minecraft evaluation, it achieves the highest combat success rate among evaluated models.
Problem
Game agents must combine high-level task reasoning with low-latency control in changing visual environments, despite the different computational profiles these capabilities favor.
Source: §1 · pp. 1–2Method
TaskFrame uses a vision-language planner to produce ordered atomic goals and passes their hidden-state embeddings to a frame-level controller that predicts native keyboard-and-mouse actions.
Source: §1 · p. 2Results
Takeaways & Limitations
TaskFrame connects explicit task decomposition to responsive frame-level control, with the large planner kept outside the per-frame loop.
Source: §1 · p. 2The agent does not decide when an atomic goal is complete or replan after failure, sees only about one second of history, and is evaluated only in Minecraft.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Building general game agents requires both high-level task reasoning and low-latency action control in rapidly changing visual environments. These capabilities favor models with substantially different computational profiles, making them difficult to realize within a single model. In this work, we present TaskFrame, a hierarchical planner-controller framework that connects task-level planning with frame-level keyboard-and-mouse control. A vision-language task planner decomposes a high-level goal into an ordered sequence of atomic instructions and is optimized through supervised learning and dependency-aware GRPO. The hidden states of special goal tokens that terminate each atomic instruction provide continuous goal embeddings that are cached and reused across control steps. A causal Transformer controller combines the embedding of the active goal with recent observations to predict discrete keyboard and mouse-button actions and continuous camera motion at every game step. We train the system in three stages: planner training, controller pretraining, and joint alignment. Experiments in Minecraft show that TaskFrame is competitive with general-purpose vision-language models on instruction decomposition, and that in closed-loop evaluation the agent achieves a higher combat success rate than the evaluated baselines.
1 Introduction
TaskFrame separates high-capacity task reasoning from low-latency frame control, connecting them through goal embeddings produced by a planner. It combines dependency-aware instruction decomposition with a controller that responds to each new visual observation, and reports strong decomposition and Minecraft combat results.
1 Introduction
- Task-level reasoning and frame-level control have different computational demands: a large reasoning model is costly at every step, while a small reactive policy lacks semantic capacity for complex instructions.Rapid visual changes also require a hybrid action space of discrete keyboard presses and continuous mouse movement.Source: §1 · p. 1
- TaskFrame uses a vision-language planner to decompose an instruction into ordered atomic goals, whose special-token hidden states condition a frame-level keyboard-and-mouse controller.The planner runs once per task; the controller combines the active goal embedding with recent frames and emits discrete actions and continuous camera motion each game step.Source: §1 · p. 2, Figure 1 · p. 2
- 51% average success rate on combat tasks, the highest among evaluated models, in closed-loop Minecraft evaluation.The 9B planner also outperforms GPT-4o and Qwen3.5-397B-A17B on instruction decomposition.Source: §1 · p. 2
- Dependency-aware GRPO rewards necessary precedence relations separately from conventional ordering to improve instruction decomposition.The planner is trained on hierarchical gameplay annotations containing atomic instructions, compositional goals, and dependency relations.Source: §1 · pp. 2–3
- The system uses a three-stage training recipe: planner training, controller pretraining with a generic goal encoder, then joint alignment through LoRA adapters.The controller has a hybrid discrete and flow-matching action head; training uses recorded human gameplay with native keyboard-and-mouse actions.Source: §1 · pp. 2–3
2 Related Work
Related work spans game agents, hierarchical task planners, and dual-system control policies. TaskFrame connects these directions through explicitly optimized ordered decomposition and latent goal-token representations passed to a fast policy.
Game agents
- Minecraft agents have conditioned low-level policies on text, videos, temporal prompts, or imagined goals, while other systems emit keyboard-and-mouse actions directly.TaskFrame instead connects explicit, trained task decomposition to frame-level control while keeping the large VLM outside the per-frame loop.Source: §2 · p. 3
Hierarchical task planning
- Prior Minecraft planners dispatch multimodal plans to goal-conditioned controllers, but are not optimized for whether plan steps respect task dependencies.TaskFrame uses must-precede annotations from GameHorizon Suite to separate necessary precedence from conventional ordering in its reward.Source: §2 · p. 3
Dual-system control
- Dual-system policies pass language, textual subtasks, or latent features between slower semantic modules and faster visuomotor policies.Like HiRT and LCB, TaskFrame passes latent goals, but its goal-token hidden states provide one embedding per step alongside an interpretable ordered plan.Source: §2 · p. 3
3 Method
TaskFrame links a planner that decomposes a high-level goal into atomic goals with a per-frame controller that uses the active goal representation to produce native actions. Training proceeds through planner optimization, controller pretraining, and planner–controller alignment.
3.1 Task Planner
- The planner is trained on aligned atomic instructions with precedence dependencies, then refined with step-count and ordering rewards that prioritize annotated dependencies over conventional order.The conventional order records a preference, whereas annotated dependencies specify must-precede relations.Source: §3.1 · pp. 4–5
3.2 Frame Controller
- The causal frame controller conditions per-frame action prediction on the planner’s goal embedding, visual tokens, and prior actions, using discrete keys and buttons plus flow-matched continuous camera motion.The action-output token cannot attend to same-frame action tokens, preventing target leakage.Source: §3.2 · pp. 5–6
3.3 Planner–Controller Alignment
- Controller pretraining uses a frozen generic vision-language goal encoder; alignment replaces it with the planner and LoRA adapters, updating the controller and adapters while freezing base planner weights.The adapted planner is used at inference for both decomposition and goal embeddings.Source: §3.3 · p. 6
3.4 Multi-Rate Inference
- At inference, the planner generates and caches goal embeddings once per high-level goal, while the controller receives the active goal embedding and emits an action at every game step.The decoded plan text is available for inspection and evaluation but is not passed to the controller.Source: §3.4 · p. 6
4 Experiments
Across Minecraft experiments, TaskFrame leads the evaluated models on combat success, performs competitively on instruction decomposition, and benefits from aligned goal embeddings and continuous camera control. The evaluations include offline action prediction and closed-loop tests that vary the supplied goal or camera head.
4.1 Setup
- The controller uses about 100 hours of gameplay with atomic-instruction conditions, while closed-loop tests cover mining and kill tasks and planner tests measure step content and ordering.Controller windows contain 21 frames, about one second at 20 Hz, and do not cross atomic-segment boundaries.Source: §4.1 · p. 7
4.2 Closed-Loop Control
- TaskFrame achieves 51% average kill success versus 31% for JARVIS-VLA-7B and is on par with it on mining, where both nearly always succeed.It has the best combat performance among evaluated models; mining is close to saturated in this setting.Source: §4.2 · p. 7, Table 1 · p. 7
- Successful rollouts show the controller approaching and attacking combat targets until killed, while mining repeatedly breaks the log ahead.Figure 3 shows four frames from each successful atomic-task episode.Source: §4.2 · p. 7, Figure 3 · p. 8
4.3 Task Planning
- The 9B planner reaches SSF1 0.3459 and Task2 accuracy 40.0%, exceeding GPT-4o and Qwen3.5-397B-A17B on both metrics but remaining below two Gemini models.The planner is evaluated on 2.1K held-out plans for SSF1 and 250 ordering questions for Task2.Source: §4.3 · p. 7, §4.1 · p. 7, Table 2 · p. 9
- The reported SFT result puts SSF1 at 0.3312 and raises Task2 accuracy from 23.2% to 38.8%; GRPO further improves both metrics and lowers the Unparsed rate from 14.0% to 11.6%.The SSF1 gain indicates GRPO does not merely fit its own reward.Source: §4.3 · p. 8
4.4 Connecting Planning and Control
- On held-out offline action prediction, TaskFrame’s aligned goal embeddings give the best results on all metrics, while oracle goals add little.Compared with the cascade, the aligned-embedding gap isolates the effect of joint alignment because both use the same decomposition.Source: §4.4 · p. 8, Table 3 · p. 9
- In closed-loop kill tasks, empty goals reduce success from 51% to 25%, mismatched goals to 13%, and withholding the planner’s observation costs 20 points.Each setting changes only the goal received by the fixed trained model.Source: §4.4 · p. 8, Table 4 · p. 9
- Flow matching lowers camera mean absolute error from 2.1° to 1.7° for yaw and from 1.2° to 1.0° for pitch versus a discrete camera head.The controllers differ only in camera head and share training data and schedule.Source: §4.4 · pp. 8–9, Table 5 · p. 8
5 Conclusion
TaskFrame separates task-level planning from frame-level keyboard-and-mouse control, using goal-token hidden states as controller embeddings. It outperforms general-purpose models on instruction decomposition and has the highest combat success rate among evaluated models, while its stated limitations include no failure-triggered replanning and Minecraft-only experiments.
5 Conclusion
- The planner outperforms GPT-4o and Qwen3.5-397B-A17B on instruction decomposition, and TaskFrame has the highest combat success rate among evaluated models.The controller relies on goal conditioning, and aligned goal embeddings are more effective than passing decoded text between modules.Source: §5 · p. 9
- The agent does not decide when an atomic goal is complete, does not replan after failure, and uses only about one second of history; experiments are limited to Minecraft.Extending the method to other games remains a challenging direction for future work.Source: §5 · p. 9
A.1 Decomposition data
- The SFT training set contains 40,068 examples: 10,442 causal, 2,914 original-segment, and 26,712 thematic.The held-out SSF1 set comprises 2,200 examples from 50 clips disjoint from the 1,119 training clips.Source: §A.1 · p. 14
A.2 Evaluation protocols
The evaluation protocols assess decomposition through semantic step matching and ordering comparisons. They also specify how examples are aggregated and how invalid or tied decisions are scored.
A.2 Evaluation protocols
- Soft Semantic F1 excludes step pairs below 0.65 cosine similarity and uses order-preserving alignment for causal examples but unrestricted Hungarian assignment for thematic examples.The score is averaged over 2,200 held-out examples, with unparseable outputs receiving zero.Source: §A.2 · p. 14
- Task2 evaluates generated plans on 250 Minecraft questions by aligning steps to canonical steps and comparing their order with four candidate permutations using Kendall correlation.Ties and questions without a valid decision count as incorrect.Source: §A.2 · p. 14
- Task2’s Unparsed rate is the percentage of questions without a valid decision, including ties.The protocol treats these cases as incorrect.Source: §A.2 · p. 14
A.3 Training hyperparameters
The appendix describes planner hyperparameters and the GRPO reward setup. Its ordering weights reflect the greater contribution of annotated dependencies than conventional ordering among free pairs.
A.3 Training hyperparameters
- The planner training hyperparameters are listed in Table 6, and the vision encoder is frozen in both training stages.GRPO uses mean-centered group rewards without dividing by within-group standard deviation, avoiding amplification of negligible reward differences when candidates score similarly.Source: Table 6 · p. 14, §A.3 · pp. 14–15
A.4 Dependency annotations and reward weights
- GameHorizon annotations indicate that thematic plans are often essentially unordered, so a single canonical-order reward would over-constrain the thematic majority.Source: §A.4 · p. 15
- Satisfying dependencies adds 30.8 points to the random-permutation ordering score, while following recorded order on remaining free pairs adds 4.6 points, motivating a 0.8/0.2 dependency/conventional-ordering split.The random permutation reaches 19.8% on the Task2 ordering criterion; the implied dependency-to-conventional contribution ratio is approximately 87:13.Source: §A.4 · p. 15
A.5 GRPO training dynamics
During GRPO training, count and order scores improve while the policy remains stable and generated plans stay below the token limit. A separate diagnostic finds naive SFT performance depends strongly on annotation-phrasing familiarity, motivating a revised SFT recipe.
A.5 GRPO training dynamics
- Count score rises from 0.630 to 0.704, while order score improves from 0.178 to 0.226–0.240 before plateauing.Table 7 reports GRPO training statistics over the first 450 steps.Source: §A.5 · p. 15, Table 7 · p. 15
- KL to the SFT policy stays between 0.02 and 0.05 after warmup, gradients show no signs of policy collapse, and plans average 73–86 tokens.Generated plans never reach the 512-token limit, so count errors are not truncation artifacts.Source: §A.5 · p. 15
A.6 Why naive SFT generalizes poorly
- On an external diagnostic, naive SFT scores 39.5% versus the backbone’s 51.2%, beating it on familiar phrasing but falling below chance on unfamiliar phrasing.The familiar-quartile scores are 76.1% versus 70.1%; the least-familiar scores are 17.0% versus 35.6%.Source: §A.6 · p. 15
- The diagnostic motivates balancing by downsampling thematic examples, describing atomic targets in two styles, and paraphrasing high-level instructions.The diagnostic used 539 four-way questions over 498 clips sharing no recordings with training data; chance was 25%.Source: §A.6 · p. 15
B.1 Training data
The controller uses annotated human Minecraft gameplay, split into instruction-consistent segments, and rebalances its long-tailed action data. Its action space combines keyboard and mouse-button predictions with continuous camera motion.
B.1 Training data
- The controller is trained on about 100 hours of first-person Minecraft gameplay, using 1,132 training clips and 35 validation clips at 20 fps.Frames are resized to 256 × 256 without cropping, and GUI frames are excluded.Source: §B.1 · p. 15
- Figure 4 summarizes per-frame action frequencies and the instruction-keyword mix of clean segments.The action frequencies cover a sample of 150 clips (785K frames).Source: Figure 4 · p. 16
- Training windows stay within clean segments, avoiding windows that straddle instruction boundaries.The split yields 33,392 segments and removes roughly 16% of otherwise eligible windows.Source: §B.1 · p. 16
- Combat makes up only 3.2% of clean segments, so training oversamples combat, placement/building, and use-action segments by factors of 6, 3, and 1.5.Moving forward accounts for 35.4% of frames and attacking for 20.9%; 54.7% of frames contain no key press.Source: §B.1 · p. 16
B.2 Action space
- The action space predicts up to four simultaneous keyboard keys and two mouse buttons, alongside camera yaw and pitch increments in degrees.Camera targets are masked on GUI frames.Source: §B.2 · p. 16
B.3 Architecture and optimization
The appendix describes auxiliary training losses, a controlled comparison of camera-motion heads, and the evaluation and baseline settings. The reported video-prediction auxiliary loss had little effect and was excluded from the method.
B.3 Architecture and optimization
- The reported controller uses an auxiliary video-prediction loss during training, but removing it had little effect and the authors do not consider it part of the method.A branch predicted eight future frames from controller hidden states and was discarded at inference.Source: §B.3 · p. 16
B.4 Flow-matching camera head
- The flow-matching camera head is compared with a discrete camera-bin head using identical controllers, data, and schedules on the held-out VPT-100h validation split.The passage identifies Table 5 as the source of results but does not provide its values.Source: §B.4 · p. 16
C.1 Task configurations
- Closed-loop tasks run in MineStudio at 256 × 256 resolution with a 400-step limit, and success is the first positive reward from the corresponding event.Mine scenes make the target block visible and reachable; kill scenes spawn four target animals 2–6 blocks ahead.Source: §C.1 · p. 17
C.2 Baseline settings
- JARVIS-VLA-7B uses its own tokenized action space and camera discretization, while UI-TARS receives 640 × 360 observations and holds each decision for 20 game steps.UI-TARS temperature is 0.7; JARVIS-VLA-7B uses 0.9.Source: §C.2 · p. 17
C.3 Per-task mining results
Mining tasks are close to saturated for TaskFrame and JARVIS-VLA-7B, while qualitative rollouts show differences on combat and mining examples. Additional evaluations describe goal-conditioning and offline action-prediction setups, and an original stone-search task exposes an exploration challenge.
C.4 Qualitative comparison with baselines
- The causal attention mask makes each frame’s action-output token attend to previous ground-truth actions, but not that frame’s action; no token attends to the action-output token.Figure 5 uses rows for query tokens, columns for key tokens, and green/gray for attended/masked positions.Source: Figure 5 · p. 18
C.3 Per-task mining results
- TaskFrame and JARVIS-VLA-7B succeed in almost every mining episode, so these tasks do not distinguish the models.Table 9 reports the per-task mining results summarized in Table 1.Source: §C.3 · p. 17, Table 9 · p. 18
C.4 Qualitative comparison with baselines
- On the illustrated kill-sheep rollout, only TaskFrame succeeds; on oak-log mining, TaskFrame, JARVIS-VLA-7B, and BAGEL succeed.Figure 6 compares same-scene rollouts, with green and red bars indicating success and failure.Source: §C.4 · pp. 17–18, Figure 6 · p. 20
C.5 Goal-conditioning ablation
- The goal-conditioning ablation keeps the trained model fixed while varying between an empty, mismatched, or visually ungrounded goal.Table 10 describes success rates over three seeds and 48 episodes per task in each seed; no cell values are supplied here.Source: §C.5 · p. 18, Table 10 · p. 19
C.6 Offline action-prediction setup
- Offline action prediction uses 50 clips disjoint from planner and controller training, with ground-truth histories and annotated boundaries for advancing atomic goals.It reports macro-F1 for key classes and mouse buttons, and camera RMSE in degrees.Source: §C.6 · pp. 18–19
C.7 Original JARVIS-VLA configurations
- Under the original JARVIS-VLA configurations, TaskFrame doubles JARVIS-VLA-7B’s zombie success but fails the stone task, where JARVIS-VLA-7B succeeds in 17% of episodes.The stone task requires locating exposed stone from a random spawn; the authors attribute the gap to exploration rather than block breaking.Source: §C.7 · p. 19, Table 11 · p. 19