PaperlayerView exact revision on arXiv

Source-linked AI summary

HiSTEVE 1.0: Harnessing Human-Interface VLA Agents for Open-World Games

Zecong Tang, Zian Guan, Yuncheng Guo, Zhanqiu Zhang, Yiwen Guo, Bo Dai

arXiv:2610.31162v1cs.CV

TL;DR

Open-world games require agents to combine long-horizon autonomy with reliable native-interface control, while existing approaches often provide one without the other. HiSTEVE pairs a capability-aware harness with a single VLA for planning and native interaction, achieving strong atomic-task benchmark results and completing a long-horizon crafting task.

  • Problem

    Minecraft agents often emphasize native visuomotor control without long-horizon task reasoning, or use high-level planning with APIs or predefined skills instead of native interaction.

    Source: §1 · p. 2

  • Method

    HiSTEVE uses a capability-aware harness for progress tracking, subtask planning, and completion verification, alongside a single VLA trained for native keyboard-and-mouse control.

    Source: §1 · p. 2

  • Results

    HiSTEVE achieves state-of-the-art performance on MineStudio Bench (85.0%) and OpenHA (37.4%) and crafts an iron pickaxe from scratch using human-interface control.

    Source: §7 · p. 10

  • Takeaways & Limitations

    The reported results indicate that reliable long-horizon execution combines regime-specific control representations with state-aware orchestration of local control.

    Source: §7 · p. 10

    Atomic subtask interfaces are manually defined, new capabilities require additional training data and policy updates, and planning depends on the selected off-the-shelf VLM backend.

    Source: H Discussion · p. 27

Abstract

from arXiv · show

Building virtual embodied agents that operate in open-ended games like human players is a long-standing goal requiring both long-horizon autonomy and reliable native-interface control. As a procedurally generated open world, Minecraft provides a natural testbed for studying the integration of these capabilities. Existing agents typically provide either native visuomotor control without long-horizon autonomy or high-level planning executed through environment APIs or predefined skills. To bridge this gap, we propose HiSTEVE, a hierarchical VLA agent with a capability-aware agent harness. At the orchestration level, we equip the harness with structured memory for tracking progress and a VLM planner for selecting subtasks within the VLA's capabilities. To reliably determine subtask completion, we further introduce a policy-coupled task completion state detector that conditions on VLA policy context to signal candidate completion. At the control level, a single VLA executes all subtasks with vision-only perception and native keyboard-and-mouse actions. To mitigate interference between heterogeneous GUI and non-GUI interactions, we introduce regime-specific RMSNorm branches and separate mouse-movement token embeddings. We also train the VLA through world knowledge injection, imitation learning, and reinforcement learning with environment feedback to improve reliability and recovery. HiSTEVE achieves state-of-the-art atomic-task performance, scoring 85.0% on MineStudio Bench and a 37.4% average success rate on OpenHA, while also completing long-horizon tasks such as crafting an iron pickaxe from scratch like human players.

1 Introduction

Open-world Minecraft agents need to combine long-horizon planning with reliable native-interface control, but existing approaches typically emphasize one or rely on APIs or predefined skills. HiSTEVE combines a capability-aware harness with a single VLA and improves benchmark performance while completing long-horizon tasks.

1 Introduction

  • Existing approaches often provide native visuomotor control without persistent task-state reasoning or planning through APIs and predefined skills rather than native interaction.Integrating task tracking, completion verification, and failure recovery remains an open challenge in the cited discussion.Source: §1 · p. 2
  • Long-horizon autonomy requires tracking progress and respecting policy capabilities, while native control must handle GUI and non-GUI mouse semantics without negative transfer.Premature handoff can leave a subtask unfinished, and late handoff can waste actions or undo progress; repeated invocation risks also accumulate.Source: §1 · p. 2
  • The agent combines structured progress memory and VLM subtask planning with a policy-coupled completion detector, visual verification, and recovery or replanning.A single VLA executes subtasks through native keyboard-and-mouse actions; regime-specific RMSNorm branches and mouse-movement embeddings address GUI/non-GUI interference, with RL intended to improve recovery.Source: §1 · p. 2
  • HiSTEVE improves average success by 20.0 percentage points on MineStudio Bench and 0.6 points on OpenHA, with LongBench family-level gains up to 28.6 points.The comparisons are against the strongest baseline in the main comparison.Source: §1 · p. 2
  • HiSTEVE achieves state-of-the-art atomic-task performance and completes long-horizon tasks such as crafting an iron pickaxe from scratch.The stated example is performed like a human player.Source: §1 · p. 3

2 Related Work

Prior work develops orchestration harnesses for coordinating planning and execution, VLA systems for mapping observations and instructions to actions, and game-playing agents for virtual worlds. These lines of work provide context for HiSTEVE's combination of long-horizon orchestration and native control.

2.1 Agent Harnesses for Execution

  • Agent harnesses evolved from tool-using language agents to systems with memory, feedback, and search, and coordinate planning with low-level execution in embodied tasks.Environmental observations, feasibility assessments, and execution outcomes can update later decisions and close the planning–execution loop.Source: §2.1 · p. 3

2.2 VLA Models and Systems

  • VLA models map visual observations and language instructions to actions, while hierarchical systems generate policy-compatible subtasks for low-level execution.More recent approaches use intermediate reasoning or predictive representations to guide low-level control.Source: §2.2 · p. 3

2.3 Game-Playing Agents in Virtual Worlds

  • Virtual-world agents span search and deep RL, pretrained instruction-following models, unified multi-game policies, and Minecraft-specific visuomotor or language-model agents.Unified multi-game policies often sacrifice control precision and long-horizon reliability.Source: §2.3 · p. 3

3 Capability-Aware Harness for Long-Horizon Autonomy

HiSTEVE's harness coordinates a VLM planner and VLA policy for long-horizon autonomy. It selects capability-compatible subtasks online, uses evidence-driven handoff and recovery, and maintains compact structured memory of verified state.

3 Capability-Aware Harness for Long-Horizon Autonomy

  • The harness coordinates a VLM planner and VLA policy through capability-constrained planning, evidence-driven handoff and recovery, and structured memory.The planner selects subtasks online rather than following a predefined route.Source: §3 · p. 3

3.1 Capability-Constrained Planning

  • Planning is reference-free: the VLM selects and instantiates an applicable parameterized subtask schema restricted to the VLA's capabilities.Schemas specify applicability, parameters, observable progress and completion conditions, failure conditions, and an execution budget; the harness turns the semantic objective into a fixed instruction for bounded VLA execution.Source: §3.1 · p. 4

3.2 Evidence-Driven Handoff and Recovery

  • The completion detector signals candidate completion or the execution budget triggers handoff; the harness then verifies visual progress and advances, recovers, or replans.The detector estimates completion probability for the current instruction; it signals candidate completion, not failure.Source: §3.2 · p. 4

3.3 Structured Memory

  • Compact memory centers on verified state transitions, using working, episodic, procedural, and world stores for current progress, invocation history, reusable dependencies, and landmarks or spatial relations.Task-relevant entries from the non-working stores are retrieved to support high-level planning.Source: §3.3 · p. 5

4 VLA Policy for Native Visuomotor Control

HiSTEVE’s VLA policy executes subtasks through native keyboard-and-mouse actions, with separate mechanisms for GUI and non-GUI interactions. Its architecture and training strategy target reliable execution across these regimes.

4.1 VLA Policy Architecture

  • The policy maps subtask instructions and first-person observations to native keyboard-and-mouse actions, generating tokenized actions autoregressively at 20 Hz.The policy is initialized from Qwen2.5-VL-7B.Source: §4.1 · p. 5, §4 · p. 5
  • RS-Norm uses interaction-specific RMSNorm scale vectors for GUI and non-GUI subtasks while keeping other parameters shared.It adds 2Ld parameters for L decoder layers and hidden size d.Source: §4.1 · p. 5

Action Representation.

  • Native actions are represented as discrete categorical groups, with mouse displacements μ-law transformed and quantized into bins.This enables supervised action prediction with the next-token prediction objective.Source: Action Representation · p. 5
  • Dual-Mouse assigns separate mouse-displacement token sets to GUI and non-GUI interactions, while both decode to the same physical action codebook.An interaction-type token identifies the current regime.Source: Action Representation · pp. 5–6

4.2 Three-Stage Policy Training

  • Training proceeds through world-knowledge injection, imitation learning, and reinforcement learning with environment feedback.Supervised stages provide Minecraft concepts with visual grounding and teach action mapping; reinforcement learning compares environment-verified branches and uses matched local preferences to target divergent action segments.Source: §4.2 · p. 6, Reinforcement Learning · pp. 6–7

Reinforcement Learning

  • Branch updates interleave with world-action replay and regularization against a fixed reference policy to preserve non-GUI capabilities.Clipped GRPO updates use informative branch groups, while DPO-style pairs target divergent segments between successful and failed branches.Source: Reinforcement Learning · p. 7

5 Task Completion State Detector

The task completion detector shares the VLA image encoder and conditions on VLA policy context to predict candidate subtask completion. It signals the harness when its probability remains above a threshold across consecutive queries.

5 Task Completion State Detector

  • The detector shares the VLA image encoder and uses policy context from the VLA’s input prefix to condition completion prediction.It processes historical frames sampled at 800, 600, 400, and 200 ms before the query, while the VLA processes the current frame and subtask instruction.Source: §5 · p. 7
  • Cross-attention injects VLA policy context into the detector after every fourth decoder layer, and activation–weight channel scores prune feed-forward width to about one third.The detector uses Qwen2.5-VL-3B as its backbone.Source: §5 · p. 7
  • A frozen-VLA-trained detector raises candidate completion only when its probability exceeds τ for two consecutive queries.Completion labels are derived from VPT human gameplay trajectories.Source: §5 · p. 7

6 Experiments

HiSTEVE is evaluated on atomic-task benchmarks, long-horizon tasks, completion detection, and ablations of its training, representation, and harness. Results show strong atomic performance, while long-horizon gains are largest on tasks requiring skill composition and persistent state tracking.

6.1 Experimental Setup

  • MineStudio Bench, OpenHA, and LongBench assess atomic and long-horizon execution, while completion detection is tested against prompted VLM and random baselines.Ablation scores are averaged over repeated independent evaluations; Table 2 uses a class-balanced validation set and shared temporal sampling.Source: §6.1 · p. 7, §6.3 · pp. 8–9, Table 2 · p. 9

6.2 Main Results

  • HiSTEVE scores 85.0% overall on MineStudio Bench and 37.4% average success on OpenHA; LongBench gains are largest in Resource & Crafting and Exploration & Memory.Table 1 compares MineStudio strict pass@1, OpenHA family-wise success and macro average, and LongBench final-goal success.Source: Table 1 · p. 8, §6.2 · p. 8
  • On MineStudio Bench, HiSTEVE raises the four-family average from 65.0% to 85.0% versus OpenHA Text/CoA, with Craft remaining its weakest category.The largest absolute gain is on Craft, where multi-step GUI manipulation remains more challenging than first-person mining and combat.Source: §6.2 · p. 8
  • LongBench final-goal success is 42.9% on Resource & Crafting, 28.6% on Exploration & Memory, and 14.3% on both Recovery and Open-ended tasks.The largest gains versus OpenHA Text/CoA occur in Resource & Crafting and Exploration & Memory; Recovery and Open-ended tasks remain harder.Source: §6.2 · p. 8

6.3 Atomic Task Completion Detection Evaluation

  • The completion detector reaches 0.863 accuracy and F1, outperforming prompted general-purpose VLM baselines and providing a more balanced candidate-completion signal.GPT-4o and GPT-5.5 reach F1 scores of 0.690 and 0.776, respectively.Source: §6.3 · p. 9, Table 2 · p. 9

6.4 Ablation Study

  • RS-Norm and Dual-Mouse jointly raise MineStudio All from 41.1% to 80.7%, with concurrent gains on OpenHA Avg. and LongBench Goal SR.The authors attribute the complementary gains to regime-specific feature scaling and mouse-action separation mitigating interference between interaction regimes.Source: §6.4 · pp. 9–10, Table 3 · p. 10
  • World knowledge injection raises MineStudio All from 50.4% to 65.1%, while environment-feedback RL raises it to 80.7% but leaves LongBench Goal SR unchanged at 14.3%.With the VLA fixed, the full harness raises LongBench Goal SR from 10.7% to 25.0%, while atomic performance changes modestly.Source: §6.4 · p. 10, Table 3 · p. 10

7 Conclusion

HiSTEVE combines a capability-aware harness with native-interface VLA control; its harness plans within the VLA's capabilities and uses verified progress to manage execution. The appendix details bounded invocations, verification and recovery, and compact structured memory.

7 Conclusion

  • HiSTEVE combines capability-constrained planning, evidence-driven handoff and recovery, and structured memory with a single VLA for native-interface control.The conclusion reports state-of-the-art results on MineStudio Bench (85.0%) and OpenHA (37.4%), and crafting an iron pickaxe from scratch using human-interface control.Source: §7 · p. 10

A.1 Harness Runtime

  • The harness repeatedly invokes a high-level planner and continuously acting VLA, updating task state at semantic execution boundaries.The planner supplies a fixed natural-language instruction for each bounded invocation; the VLA acts from changing visual observations.Source: §A.1 · p. 15

A.2 Capability-Constrained Atomic Interface

  • The planner selects from parameterized atomic-subtask schemas that expose the VLA's capabilities without providing scripted implementations.Count-based objectives use repeated bounded acquisition subtasks and update remaining requirements from visually verified progress.Source: §A.2 · pp. 15–16

A.3 Evidence-Driven Verification and Recovery

  • A detector-triggered handoff is only a candidate boundary; the harness verifies success or failure visually before advancing, recovering, or replanning.Deployment uses first-person observations and native keyboard-and-mouse actions; simulator-side state used in training is not provided to the deployed planner or VLA.Source: §A.3 · p. 16
  • Recovery handles local execution deviations; repeated failure, infeasibility, or substantial state change triggers replanning from the updated task state.Examples include temporary lack of progress, inconsistent GUI state, target loss, and an item produced but not collected.Source: §A.3 · p. 16

A.4 Structured Memory

  • HiSTEVE keeps compact memory centered on verified subtask-level transitions, using working, episodic, procedural, and world stores.Only visually verified outcomes count as successful progress, and procedural memory is updated only from verified successful executions.Source: §A.4 · pp. 16–17

A.5 Matched Harness Comparison

With the VLA and semantic model held fixed, full HiSTEVE changes orchestration and substantially improves long-horizon results while atomic-task performance changes little.

A.5 Matched Harness Comparison

  • Full HiSTEVE replaces a predetermined control horizon with state-dependent handoff, visual verification, structured memory updates, local recovery, and replanning.The fixed-horizon baseline executes the same atomic instruction for a predetermined control horizon before returning control upward.Source: §A.5 · p. 17
  • LongBench goal success rises from 10.7% to 25.0% and diagnostic progress from 17.9% to 28.6%, while MineStudio All changes from 81.2% to 85.0% and OpenHA Avg. from 37.8% to 37.4%.The controlled comparison holds the lower-level VLA and upper-level semantic model fixed.Source: §A.5 · p. 17
  • The contrast indicates that long-horizon gains arise from scheduling, monitoring, verification, and recovery over extended interaction rather than a stronger underlying VLA.The compared systems use the same controller.Source: §A.5 · p. 17

B.1 Cross-Regime Gradient Conflict

Cross-regime gradient conflict persists through training, is strongest in early decoder layers and RMSNorm components, and motivates lightweight regime-specific branches. The section also specifies the shared policy's action representation and staged training.

B.1 Cross-Regime Gradient Conflict

  • 0.1997 is the mean conflict gap across layers and components during training, with only an approximately −0.016 total change over 30,000 steps.The fitted slope is −5.4×10−7 per optimizer step; the gap rises during the first several hundred steps and persists thereafter.Source: §B.1 · p. 18, Figure 4 · p. 18
  • 0.257 is the mean gap in early layers L0–L8, versus 0.124 in deeper layers L19–L27; conflict remains positive at every decoder layer.The fitted slope across decoder depth is −0.007 per layer, and the analysis does not establish a causal source.Source: §B.1 · pp. 18–19, Figure 4 · p. 18
  • 0.606 for input RMSNorm and 0.317 for post-attention RMSNorm are the largest component conflict gaps, motivating regime-specific feature scaling.The K projection gap is 0.225; the remaining cited components have smaller gaps.Source: §B.1 · p. 19

B.2 Backbone and Regime Routing

  • Each atomic subtask's interaction regime routes to its corresponding RMSNorm branch and mouse-token set, while the remaining backbone parameters are shared.The policy retains the Qwen2.5-VL vision encoder, multimodal projector, and autoregressive decoder.Source: §B.2 · p. 19

B.3 Action Representation

  • HiSTEVE represents native keyboard-and-mouse inputs as discrete action tokens and assigns GUI and non-GUI mouse movements separate token sets despite a shared displacement codebook.Each mouse-displacement axis is independently µ-law-companded and quantized into 21 bins.Source: §B.3 · p. 19

C.1 World Knowledge Injection

  • World-knowledge injection fine-tunes the language decoder on knowledge QA, screenshot captioning, and visual question answering while freezing the vision encoder.The data are normalized, deduplicated, and quality-filtered before training.Source: §C.1 · pp. 19–20

C.2 Imitation Learning

  • Imitation learning fine-tunes the policy on gameplay and script-generated trajectories, with GUI and non-GUI examples balanced and recipe coverage augmented.The training examples pair a subtask instruction and single-frame observation with keyboard-and-mouse actions.Source: §C.2 · pp. 20–21

C.3 Reinforcement Learning

  • Reinforcement learning uses environment-state completion certificates and GUI progress rewards, alongside group-relative and pairwise objectives on rollout branches.For item acquisition, success requires the target inventory count to increase by the required amount; merely breaking a block or producing an item in a GUI is insufficient.Source: §C.3 · pp. 21–22

D.1 Dataset Construction and Evaluation Split

The completion detector is trained on trajectory-derived task labels and signals candidate completion before the VLA's next action. The harness verifies that signal, while the atomic-task evaluation uses strict first-attempt success.

D.1 Dataset Construction and Evaluation Split

  • 4,674 validation samples span mining, killing, crafting, and smelting, with samples before detected completion labeled incomplete and those at or after it labeled complete.Source trajectories are split before temporal windows are sampled, and incomplete samples are downsampled to balance the classes.Source: §D.1 · p. 22

D.2 Deployment Decision Rule

  • The detector emits candidate completion only after a two-query confirmation, and the harness then verifies visual completion conditions before updating planning state.A candidate signal returns control to the harness; it is not itself task-success evidence.Source: §D.2 · p. 23

E.1 Atomic-Task Evaluation

  • MineStudio Bench evaluates 20 atomic tasks across Mine, Kill, Craft, and Smelt, using three independent trials per task and strict pass@1.Mine and Kill primarily test first-person world interaction, while Craft and Smelt also require GUI-mediated manipulation.Source: §E.1 · p. 23

E.2 LongBench Benchmark and Evaluation

LongBench tests whether native-control skills can be composed into goal-directed behavior from visual observations and execution history. Its controlled harness comparison holds the VLA fixed, while its diagnostic patterns expose late failures in GUI execution, spatial memory, and recovery.

E.2 LongBench Benchmark and Evaluation

  • LongBench presents only a final goal and withholds reference decompositions and diagnostic annotations while agents determine intermediate objectives online.The 14 goal-only tasks cover Resource & Crafting, Exploration & Memory, Recovery, and Open-ended interaction.Source: §E.2 · pp. 23–24
  • Final-goal success is authoritative: diagnostic progress does not count as success unless the terminal task predicate is satisfied.The reported overall Goal SR is the macro-average over four task families.Source: §E.2 · pp. 24–25
  • Three recurring failure patterns are partial GUI states, exploration errors that leave spatial memory insufficient for later return objectives, and recoveries that consume the remaining execution budget.These patterns explain why trajectories can make meaningful diagnostic progress yet fail the final goal.Source: §E.2 · p. 25

E.3 Controlled Harness Evaluation

  • The controlled harness comparison holds the VLA and semantic model fixed while contrasting fixed-horizon handoff with state-dependent completion, verification, task-state updates, recovery, and replanning.This isolates runtime orchestration from changes in policy weights.Source: §E.3 · pp. 25–26, §E.2 · p. 25
  • The comparison's interpretation is that long-horizon differences primarily reflect organization of the same local controller, supported by large long-horizon changes alongside small atomic-performance changes.The supplied passage reports this interpretation without giving the controlled comparison's numerical outcomes.Source: §E.3 · p. 26

F Ablation of the State Detector Design

The detector ablations favor one-third FFN retention, sparse cross-attention, and final-layer policy-context broadcasting. The discussion reports improved long-horizon goal success with the harness and identifies capability and planner boundaries.

F Ablation of the State Detector Design

  • One-third FFN width retains 0.863 accuracy versus 0.865 at half width, while quarter width falls to 0.739; at one-third width, prefix truncation falls to 0.768.The final detector uses Wanda-inspired activation–weight scores to select retained channels.Source: F Ablation of the State Detector Design · p. 26
  • Cross-attention every two layers achieves 0.869 accuracy versus 0.859 every layer; every-fourth-layer placement uses 9 modules and remains within 0.006 of the best result.The final detector uses the every-fourth-layer configuration to balance performance and overhead.Source: F Ablation of the State Detector Design · p. 26

G Evaluation Variability

  • Table 7 reports benchmark performance as means and standard deviations on MineStudio Bench, OpenHA, and LongBench.The supplied passage identifies the table’s scope but gives no cell values.Source: Table 7 · p. 27, G Evaluation Variability · p. 27

F Ablation of the State Detector Design

  • Depth-matched intermediate policy representations reduce accuracy from 0.863 to 0.690, supporting broadcast of final-layer policy context to every cross-attention module.The final configuration supplies the same final-layer policy context to all cross-attention modules.Source: F Ablation of the State Detector Design · pp. 26–27

H Discussion

  • With the lower-level VLA fixed, the complete harness changes aggregate atomic performance modestly while substantially improving LongBench goal success.The harness preserves task state, aligns plans with executable capabilities, and adapts decisions to execution outcomes; completion signals are verified visually before state updates.Source: H Discussion · p. 27
  • The detector supplies only a candidate handoff signal; the harness verifies completion from visual evidence before updating persistent task state.This prevents individual detector errors from being directly committed to persistent task state.Source: H Discussion · p. 27
  • The system’s atomic interfaces are manually defined and its capabilities remain limited by the existing VLA; adding capabilities requires more training data and policy updates.Neither the harness nor VLA currently performs parametric continual learning from deployment experience, and planning depends on the selected off-the-shelf VLM.Source: H Discussion · p. 27
  • Future work could let the VLA learn from failure and recovery while the harness updates planning constraints and recovery strategies from experience.The paper also proposes studying transfer of planning, memory, and recovery across games and interactive domains beyond games.Source: H Discussion · p. 27

Related papers

Loading 2610.31162v1…