JARVIS-VLA: Post-Training Large-Scale Vision Language Models to Play Visual Games with Keyboards and Mouse

Muyao Li; Zihao Wang; Kaichen He; Xiaojian Ma; Yitao Liang

Source-linked AI summary

JARVIS-VLA: Post-Training Large-Scale Vision Language Models to Play Visual Games with Keyboards and Mouse

Muyao Li, Zihao Wang, Kaichen He, Xiaojian Ma, Yitao Liang

arXiv:2503.16365v2Mar 20, 2025cs.CVcs.AI

TL;DR

Prior VLA work largely emphasizes action post-training, while the paper argues that models also need environmental understanding and task-related knowledge. ActVLP adds visual-language post-training before trajectory-based imitation learning, and the authors report improved downstream imitation learning across VLM architectures, with JARVIS-VLA achieving state-of-the-art performance.

Problem
Existing VLA approaches predominantly focus on action post-training, while robust decision-making also requires environmental understanding and task-related knowledge.
Source: §1 · p. 2
Method
ActVLP post-trains VLMs on visual-language tasks before adapting them to trajectory data for action generation.
Source: §1 · p. 2, §5 · p. 10
Results
Post-training on non-trajectory tasks substantially improves downstream imitation learning across multiple VLM architectures, and JARVIS-VLA achieves state-of-the-art performance.
Source: §5 · p. 10
Takeaways & Limitations
Scaling non-trajectory vision-language tasks during post-training improves downstream task performance, according to the authors’ contribution summary.
Source: §1 · p. 2
JARVIS-VLA’s inference throughput is constrained by its large parameter size, and its performance remains below top human players’ success rates above 90%.
Source: Limitations · p. 11

Abstract

from arXiv · show

Recently, action-based decision-making in open-world environments has gained significant attention. Visual Language Action (VLA) models, pretrained on large-scale web datasets, have shown promise in decision-making tasks. However, previous work has primarily focused on action post-training, often neglecting enhancements to the foundational model itself. In response, we introduce a novel approach, Act from Visual Language Post-Training, which refines Visual Language Models (VLMs) through visual and linguistic guidance in a self-supervised manner. This enhancement improves the models' capabilities in world knowledge, visual recognition, and spatial grounding in open-world environments. Following the above post-training paradigms, we obtain the first VLA models in Minecraft that can follow human instructions on over 1k different atomic tasks, including crafting, smelting, cooking, mining, and killing. Our experiments demonstrate that post-training on non-trajectory tasks leads to a significant 40% improvement over the best agent baseline on a diverse set of atomic tasks. Furthermore, we demonstrate that our approach surpasses traditional imitation learning-based policies in Minecraft, achieving state-of-the-art performance. We have open-sourced the code, models, and datasets to foster further research. The project page can be found in https://craftjarvis.github.io/JarvisVLA.

1. Introduction

The introduction argues that action-focused post-training leaves VLA foundations under-refined, despite the importance of environmental understanding and task knowledge for flexible decision-making. It presents ActVLP and JARVIS-VLA as a response, including Minecraft instruction following across more than 1,000 atomic tasks.

1. Introduction

Imitation learning’s reliance on next-action prediction limits robust, multi-task decision-making, while action-based approaches also struggle to generalize to unseen environments or tasks.The introduction contrasts this with the more standardized nature of language tokens.Source: §1 · p. 1
ActVLP adds visual-language tasks to VLA post-training, alongside action generation, to incorporate environmental understanding and task-related knowledge.The paper proposes these capabilities as important for more flexible and generalizable decision-making.Source: §1 · p. 2
JARVIS-VLA follows human instructions across over 1,000 Minecraft atomic tasks, including crafting, smelting, cooking, mining, and killing.The paper presents this as the first VLA model in Minecraft with this instruction-following scope.Source: §1 · p. 2
Scaling non-trajectory vision-language tasks during post-training improves downstream task performance.The authors identify investigating VLA scaling laws as a contribution.Source: §1 · p. 2

2. Learning to Act from Vision Language Post-Training

ActVLP refines a VLM with world-knowledge, vision-language, and spatial-grounding tasks before adapting it to action trajectories. JARVIS-VLA combines this staged training with temporal context and an action decoder, supported by non-trajectory and gameplay datasets.

2.1. Model Structure

JARVIS-VLA uses a vision encoder, image projection module, and language model, adding observation history for partially observable settings and an action decoder for discrete and continuous actions.The action decoder represents actions as tokens in the model’s unified output vocabulary.Source: §2.1 · p. 3

2.2. Training Pipeline

ActVLP first post-trains language models on world knowledge, then jointly post-trains vision and language components on captioning, VQA, and spatial grounding, before imitation learning on trajectories.The first two stages use next-token prediction; the final stage maps instructions and observations to action chunks.Source: §2.2 · p. 4

2. Learning to Act from Vision Language Post-Training

The framework progressively refines the VLM before trajectory-based imitation learning to improve world knowledge, vision-language alignment and grounding, and action generalization.The dataset illustrations include question answering, captioning, visual grounding, and decision-making applications.Source: §2.2 · p. 4, Figure 3 · p. 5

2.3. Datasets

The non-trajectory data covers knowledge-based QA, vision-language alignment, and spatial grounding; trajectory data combines gameplay sources with synthesized expert entries for structured tasks.The paper reports over 7.4 million gameplay frames, including 6.4 million synthesized expert data entries for tasks such as crafting and smelting.Source: §2.3 · pp. 4–6

3. Experiments

The experiments evaluate JARVIS-VLA on Minecraft instruction-following tasks and test visual-language post-training, training-stage separation, and scaling. Across these tests, the authors report stronger task performance with ActVLP and improvements from scaling both trajectory and non-trajectory data.

3.1. Experimental Setup

In Minecraft 1.16.5, evaluation uses human-aligned action and observation spaces, hides unavailable player information, and tests four MCU categories with at least 30 trials per task.The categories are Mine Blocks, Kill Entities, Craft Items, and Smelt Items; Craft and Smelt require mouse-based 2D GUI manipulation.Source: §3.1 · p. 6

3.2. VLA Performance Evaluation

Over 15% higher than Qwen2-VL (IL) with 21% of its trajectory data, JARVIS-VLA-Qwen2-VL achieves the highest rates across task groups and more than double baseline success on Craft and Smelt tasks.The reported gains include more than double the baseline success rates for Craft Items and Smelt Items.Source: Table 1 · p. 7, §3.2 · p. 7

3.3. Ablation on Training Paradigm

Compared with one-stage co-finetuning, JARVIS-VLA-Qwen2-VL improved performance by approximately 57% on “Smelt iron ingot” and 23% on “Crafting the ladder” when separating visual-language post-training from action learning.Source: §3.3 · p. 8

3.4. Ablation on Non-Trajectory Datasets

Spatial-grounding post-training produces the largest downstream decision-making improvement among the tested capability variants, while each non-trajectory task type improves its corresponding VLM capability benchmark.The variants use spatial grounding, vision-language alignment, or knowledge-based question answering and share the same gameplay fine-tuning data.Source: §3.4 · pp. 8–9, Figure 4 · p. 8

3.5. Scaling Experiments

Using the same base model, increasing downstream trajectories improves VLA task success; because success is discrete, tasks show non-zero success only when evaluation loss is below 0.30.Required downstream data varies with task length and difficulty.Source: §3.5 · p. 9
For nearly all tasks, lower post-training evaluation loss correlates with higher downstream success; spatial-grounding models have the lowest loss and highest success rates.Knowledge-based models perform best for a given evaluation loss, and increasing non-trajectory data improves success with the same downstream trajectory dataset.Source: §3.5 · p. 9

4. Related Works

Prior VLA work emphasizes imitation learning and action post-training, while Minecraft VLM agents often depend on additional policies to ground actions. The paper positions its approach as enhancing the VLM backbone and generating actions directly from instructions and visual inputs.

4.1. Visual-Language-Action Models

VLA research has emphasized imitation learning and backbone selection, while few works explicitly enhance the VLM backbone through visual-language post-training for downstream VLA tasks.The cited work describes imitation learning as fine-tuning VLMs and notes prior findings on capable backbones and web-scale vision-language co-training.Source: §4.1 · pp. 9–10

4.2. VLM-based Agents in Minecraft

Minecraft VLM agents commonly use hierarchical planning and extra action-grounding policies, whereas JARVIS-VLA generates actions directly from textual instructions and visual inputs.The related-work examples include STEVE-EYE, MineDreamer, and OmniJARVIS.Source: §4.2 · p. 10

5. Conclusions

ActVLP applies vision-language post-training to improve visual-language-action decision-making in dynamic environments. The authors report downstream gains and validate the approach across multiple VLM architectures, with JARVIS-VLA achieving state-of-the-art performance.

ActVLP uses vision-language post-training to enhance decision-making capabilities of visual-language-action models in dynamic environments.Source: §5 · p. 10
Post-training on non-trajectory tasks substantially improves foundation-model understanding of complex environments and downstream imitation learning on trajectory data.Source: §5 · p. 10
The approach is validated across multiple VLM architectures, with JARVIS-VLA exemplifying state-of-the-art performance.Source: §5 · p. 10

Limitations

JARVIS-VLA remains below top human-player performance and its VLM-based parameter size constrains inference throughput. Its observation is visual-only, while its action space excludes arbitrary text input.

Limitations

JARVIS-VLA remains below top human players, whose success rates exceed 90%, and its large VLM-based parameter size constrains inference throughput.Source: Limitations · p. 11

A. Observation and Action Space

Observations use images alone, without symbolic information; the 644×364 resolution provides more visual detail than VPT’s 128×128 images.Source: A. Observation and Action Space · p. 15
The action space includes player actions except arbitrary text input, with keypresses, clicks, and discretized mouse movements represented by reserved tokens.Source: A. Observation and Action Space · p. 15, Table 3 · p. 15

B. Training Configurations

The experiments use separate training configurations for visual-language and action post-training, alongside staged action chunking and inference optimizations. Main inference results use single-action predictions for baseline comparability, and the datasets cover three non-trajectory task types.

B. Training Configurations

Visual-language post-training used a maximum token length of 3584 and 128 GPU hours, while action post-training used 512 tokens and 512 GPU hours.Source: B. Training Configurations · p. 15

C. Details of Inference

The inference example predicts camera movements and then a use action in tokenized action sequences to carry out a bread-crafting instruction.Source: C. Details of Inference · p. 16

Action Chunking Strategy

The 1→3→2 chunking schedule achieved the highest success rate (0.67) and 15 FPS in the reported ablation; main experiments used one inference action for fair baseline comparison.Source: Action Chunking Strategy · p. 17, Table 4 · p. 17

Infrastructure for Efficient Inference

vLLM increased inference throughput by up to 1.7 times without significant task-performance degradation.Source: Infrastructure for Efficient Inference · p. 17

Achieved Inference Performance

JARVIS-VLA-Qwen2-VL-7B runs at 5.5 FPS and JARVIS-VLA-Qwen2-VL-2B at 8 FPS with single-action inference; the 2B model reaches 15 FPS with chunk size two.Source: Achieved Inference Performance · p. 17

D. Datasets

The visual-language post-training datasets comprise world knowledge, vision-language alignment, and spatial grounding data.Source: D. Datasets · p. 17

D.1. World Knowledge Dataset

The paper describes three non-trajectory dataset types and their construction, then outlines a benchmark spanning world knowledge, visual recognition, and spatial grounding. Spatial-grounding test data is collected to avoid overlap with training data.

D.1. World Knowledge Dataset

The world-knowledge dataset draws on Wikipedia and DigMinecraft content, uses GPT-3.5-turbo-generated answers, and organizes examples into multi-turn dialogues of about 1500–2000 tokens.Source: §D.1 · pp. 17–18

D.2. Vision Language Alignment Dataset

The vision-language alignment dataset uses 35,000 selected keyframes to produce 15,000 captions and 20,000 visual question-answer pairs, with a validation pipeline filtering ambiguity and checking answer accuracy and image relevance.Source: §D.2 · p. 19

D.3. Visual Grounding dataset

Visual grounding data combines backward-tracked 3D object positions and GUI item locations, yielding 404K point data for entity and object positions.Source: §D.3 · p. 20

E. Benchmarks

The benchmark covers world knowledge, visual recognition, and spatial grounding; spatial-grounding test data is collected so it does not overlap with training data.Source: E. Benchmarks · p. 21

E.1. World Knowledge Evaluation

The World Knowledge Evaluation organizes Minecraft knowledge questions into eight subcategories and includes 47 test questions.

E.1. World Knowledge Evaluation

The evaluation covers craft, plant, mobs, kill, enchant, potion, biomes, and architecture.Source: §E.1 · p. 21
The benchmark contains 47 test questions, with a correct answer provided for each entry.Source: §E.1 · p. 21
Table 5 provides a detailed description of the knowledge questions and answers.Source: Table 5 · p. 21

E.2. Vision Understanding Evaluation

The paper evaluates visual understanding with image-based questions and spatial grounding with object-location outputs. It also illustrates model responses to a question about the time of day.

E.2. Vision Understanding Evaluation

Vision understanding comprises 43 questions across scene, object, info, inventory, and OCR categories.Scene, object, and info questions cover the game environment, visible objects, and player status, respectively.Source: §E.2 · p. 22, Table 6 · p. 22
For the time-of-day example, JARVIS-VLA answers night, while GPT-4o is uncertain and Qwen2-VL answers daytime.Source: §E.2 · p. 23

E.3. Spatial Grounding Evaluation

Spatial grounding uses 100 GUI entries and 236 embodied entries, with point locations preferred and bounding boxes used when points are unavailable.Outputs are normalized to [0, 1000).Source: §E.3 · p. 23, Table 7 · p. 23

E.4. Evaluation Metric and Result

The evaluation uses LLM judging for Minecraft knowledge and visual-understanding answers, and rule-based scoring for visual grounding. Table 8 compares several VLMs and reports that the method improves core capabilities while leaving a gap to state-of-the-art models.

E.4. Evaluation Metric and Result

GPT-4o judges knowledge and visual-understanding responses against expert reference answers as correct or incorrect, while visual grounding is scored by rules.Source: §E.4 · p. 23
Table 8 reports that the method significantly enhances models’ core capabilities, though a gap remains relative to state-of-the-art models.Source: Table 8 · p. 24
Table 8 compares JARVIS-VLA with commercial and open-source models, including GPT-4, GPT-4-mini, Llava-Next, Molmo-d-0924, Llama-3.2, and Qwen2-VL.Source: Table 8 · p. 24, §E.4 · p. 24

F. Ablation with different Pre-trained VLMs

The ablation compares ActVLP across two VLM backbones with different pretraining datasets and image-processing techniques. Both show downstream task success-rate increases exceeding 30%.

F. Ablation with different Pre-trained VLMs

The comparison uses Llava-Next and Qwen2-VL, which differ in pretraining datasets and image-processing techniques.Source: F. Ablation with different Pre-trained… · p. 24
More than 30% increases in downstream task success rates follow ActVLP post-training for both Llava-Next and Qwen2-VL.The authors describe the improvement as robust across the two models.Source: F. Ablation with different Pre-trained… · p. 24
Table 9 defines raw as directly fine-tuned Qwen2-VL, RT-2 as one-stage co-finetuned Qwen2-VL-7B, and Act-VLP as the three-stage post-training approach.Source: Table 9 · p. 24