Vision-Language Models Provide Promptable Representations for Reinforcement Learning

• RL agents need contextual representations that can index vast world knowledge into grounded features useful for acquiring new behaviors.Source: §1 · p. 1
• PR2L avoids both ungrounded action generation and generic image encoding by using prompted VLM representations for low-level control.Source: §1 · p. 2
• PR2L prompts VLMs to produce task-specific semantic features that integrate visual observations with prior knowledge and are grounded into actions through RL.The decoded text is discarded; associated embeddings feed a learned policy.Source: §1 · p. 2
• Across Minecraft and Habitat, PR2L outperforms vision-only and instruction-conditioning approaches and is competitive with domain-specific representations.The evaluation covers visually complex tasks in both online and offline RL settings.Source: §1 · p. 2

Vision-language models

• Generative VLMs model language responses conditioned on images and prompts, with Transformer representations depending on visual inputs, prompts, and preceding tokens.Source: §2 · p. 2

Embodied (V)LM reasoning

• Plan-oriented embodied VLM systems extract textual subtasks or code but often require grounding mechanisms and pre-existing low-level skills.They generally do not explain how those skills are acquired or how they benefit from VLM knowledge.Source: §2 · p. 3
• PR2L differs by extracting task-relevant world knowledge into representations and training a policy to convert them into low-level actions via RL.Source: §2 · p. 3

Embodied (V)LM pre-training

• VLM encoders have been used for instructions, feedback, reward specifications, and world-modeling data, primarily to transfer compositional semantic structure.Source: §2 · p. 3
• PR2L asks VLMs to parse observations into task-relevant semantic features, rather than merely conditioning policies on instructions.Source: §2 · p. 3

• Related instruction-following approaches largely focus on manipulation and behavior cloning, whereas PR2L supports both online RL and offline RL or BC.Source: §2 · p. 3

3 PR2L: Promptable Representations for Reinforcement Learning

• PR2L queries a VLM for each visual observation and feeds its intermediate promptable representations, rather than decoded text, into an RL policy.The method does not require VLM fine-tuning or direct action generation.Source: §3 · p. 3, §3.1 · p. 4

3.1 Promptable Representations

• Direct VLM action generation is poorly suited to common low-level RL actions, so PR2L elicits semantic observation features that policies can link to actions.Source: §3.1 · pp. 3–4
• A Transformer layer with a learned CLS token condenses variable-length VLM representations into a fixed-length summary for the policy.Source: Figure 2 · p. 4, §3.2 · p. 5
• Transformer token embeddings from the prompt, image, and generated response encode task-relevant information and become the policy state representation.Source: §3.1 · p. 4

3.2 Design Choices for PR2L

• Greedy decoding or fixed-seed sampling is preferred because tasks are high-variance and VLM decoding is expensive; later layers are used for semantic information.Source: §3.2 · pp. 4–5

3.3 Task-Relevant Prompt Design

• PR2L uses task-relevant questions, optional auxiliary information, and chain-of-thought prompting to make VLM representations attend to useful semantic features.Source: §3.3 · p. 5
• Task-relevant prompts differ from instructions because they produce state representations for RL rather than directly requesting optimal actions.Source: §3.3 · p. 5
• Candidate prompts are evaluated on labeled semantic observations or with small probes, providing a cheaper proxy for downstream task performance than running RL for every prompt.Source: §3.3 · p. 5

4.1 Domain 1: Minecraft

• Minecraft evaluation covers six long-horizon programmatic tasks using egocentric RGB observations, pose, previous action, and discrete movement, attack, and item-use actions.Episodes last up to 500–1000 steps, and learned policies take roughly 200 steps to complete tasks.Source: §4.1 · p. 6

4 Experimental Setups

• PR2L is evaluated against non-promptable visual embeddings, direct VLM action generation, Dreamer, control-specific encoders, and Minecraft-specific oracle representations.The comparisons use matched policy architectures where applicable and distinguish general-purpose frozen VLM representations from Minecraft-trained oracles.Source: §4.1 · p. 6

4.2 Domain 2: Habitat

• Habitat evaluation uses ObjectNav in 80 training and 20 unseen validation household scenes, with RGB vision, previous action, pose, and target object class but no depth.The tasks require finding household objects while traversing scanned 3D environments, testing generalization in naturalistic scenes.Source: §4.2 · pp. 6–7
• Habitat policies are trained on just under one-tenth of the available 77k-trajectory, 12M-step dataset, limiting the training-data scale relative to prior work.Evaluation still uses unseen validation scenes to test generalization.Source: §4.2 · p. 7

4.3 Designing Task-Specific Prompts for Minecraft and Habitat

• Prompt design uses target-entity presence questions in Minecraft and location-based questions in Habitat to elicit task-relevant semantic features.Habitat prompts ask whether an object would be found in the observed location and can request a chain-of-thought rationale based on common-sense knowledge.Source: §4.3 · pp. 7–8
• Auxiliary prompt text helps spider detection but degrades sheep and cow detection, and prompt-detection success correlates with downstream RL performance.The prompt-selection procedure evaluates candidate prompts on a small annotated dataset rather than running full RL for every candidate.Source: §4.3 · p. 7
• PR2L universally outperforms baselines in Minecraft, while Minecraft-specific oracle representations outperform PR2L in half the comparisons.Table 2 reports IQM successes and standard errors across 16 seeds, with oracle methods benefiting from Minecraft-specific training data.Source: Table 2 · p. 8
• PR2L with CoT achieves the best Habitat ObjectNav performance, more than doubling all non-PR2L approaches and attaining 14.7% higher average performance than PR2L without CoT.Values are average success rates in unseen validation scenes.Source: Table 3 · p. 8

Minecraft results

• PR2L uniformly outperforms non-promptable image embeddings, direct VLM action generation, learning-from-scratch Dreamer, and control-specific embeddings on Minecraft tasks.Minecraft results report interquartile means and standard errors over 16 seeds.Source: §5 · p. 8
• PR2L exceeds the VLM image-encoder baseline despite receiving the same visual features, indicating that prompting beneficially transforms representations for control.The comparison uses no additional environmental information.Source: §5 · p. 8

Ablations

• PR2L performs worse without prompting or text decoding, while prompt evaluation identifies when auxiliary information helps or harms performance.An oracle entity detector also performs worse than PR2L, suggesting the prompt elicits more than entity detection alone.Source: §5 · p. 8

Habitat results

• 41.9% success rate with CoT versus 27.8% without CoT shows a 1.5× improvement on Habitat ObjectNav.PR2L with CoT outperforms all other policies on every task and is nearly four times better than VLM image-encoder baselines.Source: §5 · p. 9
• PR2L does not exceed VC-1’s reported 60.3% success rate, but reaches strong performance with comparable training data and fewer computational demands.VC-1’s result used over ten times more training data and gradient steps plus image augmentations, whereas PR2L generalizes to unseen scenes without explicit overfitting countermeasures.Source: §5 · p. 9

Offline RL

• Offline-RL analyses find that VLM representations align well with the returns of an optimal policy in a simplified Habitat setting.This analysis is reported in Appendices D and H.Source: §5 · p. 9

Conclusion

• PR2L extracts semantic image features by prompting VLMs with task context and uses their general-purpose prior knowledge for RL control.The method is demonstrated in Minecraft and Habitat, where observations benefit from semantic interpretation.Source: §6 · p. 9

Limitations

• Prompts are currently hand-crafted around the user’s conception of useful task features, although their evaluation and improvement could be automated.The authors leave prompt automation to future work.Source: §6 · p. 9
• Representation quality depends strongly on the VLM, while model size and speed can limit applicability.The policies typically run at 3–5 Hz, and offline training can mitigate inference-speed constraints.Source: §6 · p. 9

A Prompt Evaluation for RL in Minecraft

• The evaluation collects human-recorded Minecraft videos showing target entities left, right, middle, or absent across varied views.Videos generally last about 30 seconds at 30 frames per second, with the absent-entity video lasting one minute.Source: A Prompt Evaluation for RL in Minecraft · p. 15
• Prompts test entity presence directly through yes/no answers and entity position through a three-category linear classifier over decoded-token embeddings.The classifier measures whether position is easily extractable from representations even when text outputs fail to distinguish positions.Source: A Prompt Evaluation for RL in Minecraft · p. 15
• Auxiliary visual information greatly helps the Minecraft spider case but excessive prompt detail degrades performance, likely because longer prompts are out of distribution.Cows and sheep more closely resemble their natural counterparts, so added information is less helpful for them.Source: A Prompt Evaluation for RL in Minecraft · pp. 15–16
• Figure 3 presents example Minecraft and Habitat tasks, observations, and task-relevant prompts used to contextualize the evaluation.These examples connect visual observations to the prompts supplied to the VLM.Source: Figure 3 · p. 16
• All three candidate prompts yield similarly high linear classifiability of target position in their decoded embeddings.Thus, representations retain granular positional information even when the prompt does not explicitly request it.Source: A Prompt Evaluation for RL in Minecraft · p. 16, Figure 4 · p. 17

B.1 Environment Details

• The Minecraft agents use egocentric RGB observations, pose, previous action, and discrete action spaces of 53 or 89 actions.The observation space omits nearby voxel labels and limits pitch for combat spider; turning and movement comprise the main action groups.Source: §B.1 · pp. 16–17
• Minecraft tasks include combat, milking, and shearing objectives with task-specific rewards, episode lengths, distractors, and entity configurations.Combat spider includes distractor animals and rewards strikes and defeat; milk cow and shear sheep reward approach and successful interaction, while zombie, enderman, and pigman variants alter episode or enemy settings.Source: §B.1 · p. 17
• The reported Minecraft experiments share PPO and policy hyperparameters across baselines, PR2L, and ablations, with smaller model dimensions for some combat tasks.Tables 6 and 7 summarize these settings.Source: Table 6 · p. 18, Table 7 · p. 18

B.2 Policy and Training Details

• Training uses Stable-Baselines3 clipped PPO, with a cosine learning-rate schedule for spider trials and deterministic observations, actions, and dynamics in the described setting.Source: §B.2 · p. 19, §C.1 · p. 19

C.1 Environment Details

• Habitat ObjectNav uses RGB observations, agent pose, compass, previous action, and a one-hot object goal, while depth is removed to emphasize visual grounding.Source: §C.1 · p. 19
• The action space contains four actions: turning by 30 degrees, moving forward, stopping, and the remaining standard Habitat action.Source: §C.1 · p. 19
• Agents navigate household environments toward specified target objects, requiring common-sense knowledge about typical object locations and home structure.Source: §C.1 · p. 19
• Habitat provides 80 training scenes and 20 novel validation scenes from the Habitat-Matterport 3D v1 dataset to assess generalization.Source: §C.1 · p. 20

C.2 Policy and Training Details

• The main Habitat training set contains 1.1M steps over 7,550 trajectories, sampled because compute limits prevented using the full 77k-trajectory demonstration dataset.Source: §C.2 · p. 20
• Full-trajectory batches can reduce scene diversity because of compute constraints, although they enable up-to-date recurrent hidden states.Source: §C.2 · p. 20
• The policy generalizes strongly to unseen validation scenes despite using neither image augmentation nor loss regularization.Source: §C.2 · p. 20
• PR2L uses Prismatic VLM representations, dimensionality reduction, final-layer embeddings, sample-based decoding, and a learned Transformer policy layer.Source: §C.2 · pp. 20–21

D. Simplified Habitat Offline RL Experiments

• The simplified offline ObjectNav benchmark uses 32 reconstructed homes containing toilet, bed, and sofa goals, with decisions based only on current vision and pose.Source: §D.1 · p. 21

D.2 Policy and Training Details

• Offline RL uses Conservative Q-Learning with QR-DQN, selecting evaluation actions by the highest mean quantile value.Source: §D.2 · pp. 21–22

D.3 Experiments and Results

• The baseline uses InstructBLIP’s unprompted Q-Former embeddings and the prompt “What room is this?” to control embedding-sequence length.Source: §D.3 · p. 22
• PR2L outperforms the baseline for final evaluation success rates and average returns both per target object and overall.Source: Figure 5 · p. 22
• 60.4% versus 35.2%: PR2L nearly doubles the baseline’s average success rate in simplified Habitat ObjectNav.The comparison is reported alongside evaluation success rates and average returns in Figure 5.Source: §D.3 · p. 22, Figure 5 · p. 22
• PR2L representations correlate with an expert policy’s value function, with high-value states typically labeled as rooms where the target object is expected.Source: §D.3 · p. 22

E.1 Notes on Task-specific Systems

• The comparisons exclude systems targeting specialized task infrastructure or higher-level plans because PR2L is presented as a general policy-learning approach.Source: §E.1 · pp. 22–23
• Voyager’s harder-task performance relies on hand-written primitives, including oracle pathfinding and attack controllers, rather than learning those primitives with a VLM.Source: §E.1 · p. 23
• PR2L learns policies linking observations directly to low-level actions without oracle information, unlike systems that compose hand-crafted control primitives.Source: §E.1 · p. 23

E.2 Notes on Dreamer v3

• PR2L does not prescribe a particular learning algorithm, so combining its representations with Dreamer remains future work.Source: §E.2 · p. 23
• Dreamer v3 fails to learn the six Minecraft tasks under the matched interaction budget, which the authors hypothesize relates to visual reconstruction with partially observable, non-stationary entities.Source: §E.2 · p. 23

F Ablations

• PR2L especially outperforms the VLM image-encoder baseline on combat spider, even after one training epoch.Source: Figure 6 · p. 24
• All ablations perform worse than PR2L, supporting the importance of both promptable and generative VLM representations for control.The ablations remove prompting, generation, prompt auxiliary information, or add an oracle entity detector.Source: F Ablations · pp. 23–24
• For milk cow, the no-generation variant performs best among ablations, possibly because generated text is often wrong and auxiliary information further reduces detection quality.Source: F Ablations · p. 24
• Prompt evaluation serves as an effective proxy for selecting representations, because prompts predicted to be weaker also produce worse downstream performance.Source: F Ablations · p. 24

G Minecraft Behavior Cloning Experiments

• Behavior cloning trains policies on successful expert trajectories embedded with either PR2L or the VLM image encoder, then evaluates task success rates.Source: G Minecraft Behavior Cloning Experiments · p. 24
• PR2L achieves approximately twice the combat-spider success rate of the VLM image-encoder policy after one training epoch.Both shear-sheep policies are weak because the expert data came from a MineCLIP-trained policy, limiting that task’s effectiveness.Source: G Minecraft Behavior Cloning Experiments · p. 24, Figure 7 · p. 25
• The combat-spider result supports the hypothesis that promptable representations help link observations to appropriate actions in offline learning rather than solving exploration.Source: G Minecraft Behavior Cloning Experiments · p. 24, H Representation Analysis · p. 25

H Representation Analysis

• The analysis asks whether task-relevant representations organize observations around rewarding functional actions more effectively than instruction prompts.Source: H Representation Analysis · p. 25

H.1 Minecraft Analysis

• PCA projects tokenwise-averaged embeddings from expert rollouts into two dimensions, with colors marking functional versus movement actions and larger points marking rewarded observations.Source: §H.1 · p. 25
• Rewarding functional actions cluster with VLM “yes” outputs under task-relevant prompts, whereas instruction-prompt representations distribute them more broadly.This structure is especially clear for milk cow, where rewarding bucket-use actions concentrate in the lower-left cluster.Source: §H.1 · pp. 25–26

H.2 Habitat Analysis

• In Habitat, high-value states group with semantically appropriate rooms—bathrooms for toilets, bedrooms for beds, and living rooms for sofas—unlike generic image-encoder features.The PR2L representation structure therefore correlates with the value function of a near-optimal policy.Source: §H.2 · pp. 26–27, Figure 8 · p. 26

I Code Snippets

• The policy projects VLM embeddings, condenses them with a transformer and CLS token, and predicts actions through a linear action head.Source: I Code Snippets · p. 27
• The code path feeds an image and task-relevant prompt to a generative VLM, extracts image, prompt, and decoded-token embeddings, and maps them through a policy to action logits.Source: I Code Snippets · pp. 28–29

J Extended Literature Review

• Prior embodied foundation-model work uses models for plans, instructions, feedback, rewards, or world modeling, whereas PR2L uses VLM representations for atomic-task RL.Source: J Extended Literature Review · p. 29