PaperlayerView exact revision on arXiv

Source-linked AI summary

Selective Agent Guidance via Entropy: Learning Autonomous Policies from Imperfect VLM Teachers

Giovanni Bonetta, Matteo Merler, Davide Zago, Rossella Cancelliere, Bernardo Magnini

arXiv:2609.01567v2cs.AIcs.CLcs.LG

TL;DR

Direct VLM policies are costly and brittle, motivating the question of how to learn a cheap autonomous policy from an imperfect online teacher. SAGE queries the VLM selectively during uncertain training states, executes and advantage-weights its guidance within RL, and distills it into a policy that acts without VLM calls at evaluation. Across sparse-reward visual tasks, SAGE improves over unguided RL in several environments and can produce policies that outperform the guiding VLM, although uninformative or misleading teachers can cause negative results.

  • Problem

    Direct VLM policies require costly per-step querying, do not improve through environment interaction, and may repeat systematic errors.

  • Method

    SAGE queries a VLM when learner policy entropy indicates uncertainty, executes the teacher’s action during training, and distills guidance using environment-derived advantages.

  • Results

    Across sparse-reward visual decision-making tasks, SAGE improves over unguided RL in several environments and can learn autonomous policies that outperform the guiding VLM.

  • Takeaways & Limitations

    VLMs can serve as temporary, imperfect training guidance whose value is tested through interaction and internalized by a lightweight autonomous policy.

  • Takeaways & Limitations

    SAGE assumes the teacher retains task-relevant competence; uninformative or systematically misleading guidance can cause negative results, and safety is not guaranteed under misspecified or incomplete rewards.

Abstract

from arXiv · show

Vision-Language Models (VLMs) provide useful priors for interactive decision-making, but using them directly as policies is expensive and brittle: they must be queried at every step, do not improve from environment interaction, and can repeat systematic errors. We study how to learn a cheap autonomous policy from an online, expensive, and imperfect but informative VLM teacher. We propose SAGE (Selective Agent Guidance via Entropy), a framework that queries a VLM only when the learner is uncertain, executes the suggested action during training, and distills guidance into a lightweight Reinforcement Learning (RL) policy. Because VLM advice is not always reliable, SAGE can weight teacher-action distillation using environment-derived advantages rather than treating all suggestions as equally useful. Across sparse-reward visual reasoning and navigation tasks, SAGE learns policies that act without VLM guidance at evaluation time and improves over unguided RL in several environments, including settings where the learned policy exceeds its VLM teacher. The results show that selective guidance is most beneficial when the VLM can help the agent discover high-reward trajectories, and less useful when unguided exploration already succeeds or teacher actions do not lead to informative experience. SAGE also reduces VLM usage by prompting the teacher only on a fraction of training steps and requiring no VLM calls at deployment. Overall, our results suggest that VLMs don't need to be used as fixed policies to be useful; they can instead act as temporary, imperfect sources of guidance whose value is tested and internalized through interaction.

1 Introduction

SAGE treats an imperfect VLM as a selective training-time teacher for a lightweight RL policy, addressing sparse-reward exploration while avoiding costly deployment-time querying. It evaluates whether guidance can improve autonomous policies across visual navigation, reasoning, and interaction environments.

  • Motivation: Sparse rewards make trial-and-error inefficient when successful visual, symbolic, or multi-step trajectories are rarely discovered.The problem is especially acute when agents must ground perception, reason symbolically, and complete several correct actions before receiving informative reward.
  • Motivation: Directly deploying a VLM is expensive and slow because it requires prompting at every decision step and can repeat systematic mistakes.A frozen VLM also cannot improve through environment interaction.
  • Method: SAGE queries a VLM when the learner’s policy entropy exceeds a threshold, executes the suggested action, and distills guidance into the RL policy.The framework uses entropy as a lightweight uncertainty proxy and can weight teacher actions by environment-derived advantages.
  • Evaluation: Evaluation covers sparse-reward visual navigation, object interaction, arithmetic reasoning, perceptual symbol matching, and grounded household interaction without VLM guidance at test time.Held-out environment seeds measure what the lightweight policy internalized rather than online VLM assistance.
  • Results: SAGE can improve over PPO where exploration bottlenecks dominate, including cases where the learned policy exceeds its VLM teacher.Compared with full-query approaches, it queries the VLM on only a fraction of training steps and requires no deployment-time calls.

2 Related Work

Related work uses imitation, uncertainty-triggered supervision, foundation models, and imperfect-demonstration weighting to support interactive learning. SAGE combines selective online VLM action guidance with environment-derived advantage weighting and autonomous deployment.

  • Imitation learning and online supervision: Online supervision methods query experts during learner execution, while SAGE uses an expensive and imperfect VLM and therefore selects interventions selectively.This contrasts with approaches that assume a reliable expert or query the expert without the same teacher-quality assumption.
  • Learning from imperfect demonstrations: AWR, AWAC, and CRR weight prior actions by estimated value so suboptimal behavior can aid learning without being cloned uniformly.SAGE applies this principle to online VLM-guided RL using environment-derived advantages.
  • Foundation models for interactive decision-making: Foundation models have been used as feedback sources through preferences, rewards, code, embeddings, learned reward models, or direct policies.These approaches position foundation models as learning signals or deployed decision-makers rather than only selective teachers.
  • Distillation into smaller policies: Distillation methods such as LM4TEACH and LVLM2P transfer foundation-model supervision into smaller policies, but differ from SAGE in supervision format and query frequency.SAGE asks for a single action, queries only in high-entropy states, and trains the final policy to act without the teacher at evaluation.

3 Method

SAGE selectively queries an online VLM teacher when the learner is uncertain, separates teacher-guided transitions from on-policy learning, and distills guidance using environment feedback. This design addresses off-policy mismatch and imperfect teacher actions while retaining autonomous deployment.

  • Entropy-Gated Selective Guidance: SAGE uses normalized policy entropy as an uncertainty proxy, querying the VLM when entropy exceeds threshold ν and otherwise sampling from the learner.The teacher’s suggested action is executed during training, while a cache can avoid repeated prompts for identical states.
  • Partitioned Loss Functions: Teacher-guided actions are excluded from PPO because they can create an off-policy mismatch and unstable importance ratios when the learner assigns them low probability.The partitioned objective addresses this mismatch by keeping teacher actions out of the clipped PPO policy loss.
  • Partitioned Loss Functions: Transitions are partitioned into learner-generated Bπ and teacher-guided BT subsets, with PPO applied only to Bπ and behavioral cloning applied to BT.This preserves PPO’s on-policy interpretation while allowing teacher actions to influence the policy through a separate supervised objective.
  • Partitioned Loss Functions: The value function trains on the full experience buffer using environment rewards, including teacher-guided transitions.Such transitions may expose successful trajectories that are rarely discovered through unguided exploration in sparse-reward tasks.
  • Advantage-Weighted Behavioral Cloning: Advantage-Weighted Behavioral Cloning scales teacher-action distillation using critic-derived advantages, emphasizing actions associated with higher-return trajectories and down-weighting poorer outcomes.Weights use an exponential advantage transform with temperature τ and are clipped to 20 for stability.
  • Advantage-Weighted Behavioral Cloning: AWBC is treated as an optional refinement because the critic may be uninformative early in training, when it behaves similarly to unweighted behavioral cloning.Blindly cloning incorrect VLM actions can reinforce mistakes and reduce future uncertainty, motivating empirical evaluation of the weighting scheme.

4 Experimental Setup

SAGE is evaluated across sparse-reward visual navigation, interaction, arithmetic reasoning, and perceptual symbol-matching environments. Experiments compare unguided PPO, direct VLM control, VLM-assisted RL methods, ablations, and oracle-guided variants.

  • Environments: FrozenLake uses procedurally generated 8×8 maps and rewards success only, preventing memorization of a fixed route.The task requires reaching the goal while avoiding holes under sparse rewards.
  • Environments: The benchmark suite includes FrozenLake, three MiniGrid tasks, EZPoints, CardMaze, and ALFWorld, covering navigation, object interaction, arithmetic reasoning, symbol matching, and household interaction.CardMaze is introduced as a novel perceptual symbol-matching environment with reward after five consecutive correct selections.
  • Environments: ALFWorld combines visual observations, text goals, and admissible text actions, and is reported as an exploratory stress test because of simulator and VLM inference costs.
  • Baselines: SAGE is compared with standard PPO, direct VLM control, and VLM-assisted RL baselines including RL-VLM-F and LVLM2P.The comparisons test reward shaping, supervised policy distillation, and direct VLM action selection against SAGE’s guidance mechanism.
  • Ablations: Ablations remove behavioral cloning or advantage weighting to test the roles of guided exploration, explicit teacher-action supervision, and environment-derived weighting.
  • Ablations: Oracle variants replace the VLM with a rule-based teacher to diagnose whether performance is limited by teacher quality or the learning framework.The oracle is intended as a diagnostic rather than a deployable baseline.

5 Results

SAGE improves autonomous performance and reduces VLM dependence in settings where guidance helps overcome exploration bottlenecks, but benefits vary with teacher quality and baseline competence.

  • Main Results: SAGE improves over PPO in several environments, with clearest gains on CardMaze, GoToDoor, Fetch, and exploratory ALFWorld.Remaining environments expose limits from teacher quality or already-strong baselines.
  • Main Results: On CardMaze, VLM-as-policy scores 0.000 while SAGE reaches the optimal return of 1.000, exceeding its direct VLM teacher.DAgger reaches 0.993 but queries the VLM at every training step.
  • Main Results: SAGE queries the VLM on only 1.2%–13.3% of training steps across six controlled environments, versus 100% for full-query baselines, and uses no VLM calls at deployment.Queries concentrate early in training and are highest on CardMaze at 13.3%.
  • Main Results: SAGE uses selective exploration interventions and environment feedback, whereas RL-VLM-F, LVLM2P, and DAgger struggle with consistent cross-environment gains for different supervision or output-format reasons.SAGE asks for a preferred action and distills it using policy learning shaped by environment rewards.
  • Main Results: SAGE is not uniformly beneficial: PPO already solves LavaGap, direct VLM control is stronger on Fetch, and Qwen guidance performs poorly on EZPoints.The results characterize teacher quality and existing exploration ability as practical boundaries on gains.
  • Effect of Teacher Quality: Teacher usefulness is not predicted by direct-policy performance alone: Gemma enables SAGE to reach 10.000 on EZPoints, while Qwen-guided SAGE remains at 0.000.On CardMaze, either teacher yields optimal SAGE performance despite differing direct-policy returns.
  • Ablation Study: Removing explicit behavioral cloning collapses performance even when teacher actions are executed, while advantage weighting shows no consistent benefit over plain BC.The ablation identifies explicit BC distillation as essential and AWBC as optional under these experiments.
  • Long-Horizon Evaluation: After 5M steps, PPO remains near zero on FrozenLake, EZPoints, and CardMaze, whereas SAGE + Oracle reaches near-optimal performance, showing guidance changes discovered trajectories.On MiniGrid, PPO is stronger, although oracle guidance still improves Fetch and GoToDoor.

6 Conclusions

SAGE treats VLMs as temporary, imperfect exploration priors rather than deployed policies, selectively distilling useful guidance into lightweight autonomous RL agents. Across sparse-reward visual decision-making tasks, it improves over unguided RL in several environments and can produce policies that outperform their guiding VLMs.

  • SAGE learns lightweight RL policies from an online, expensive, and imperfect but informative VLM teacher.The VLM is used as a temporary exploration prior rather than as the deployed policy.
  • During training, SAGE queries the VLM in high-entropy states, executes teacher actions, and distills useful guidance through an RL objective with optional advantage weighting.This design lets environment interaction determine which teacher actions are reinforced.
  • SAGE improves over unguided RL in several sparse-reward visual environments and can learn autonomous policies that outperform the VLM used for guidance.The conclusion identifies this pattern across the evaluated visual decision-making tasks.
  • Direct VLM-policy performance alone does not determine teacher usefulness, because even weak direct policies can provide interventions that help agents discover high-reward behavior.The paper therefore supports using VLMs as temporary exploration guides for autonomous RL agents.

Limitations

SAGE’s scope and reliability are constrained by its entropy-based query rule, discrete-action experiments, dependence on task-relevant teacher competence, and preliminary embodied evaluation. The paper also cautions that distillation can transmit unsafe or biased teacher behavior when rewards are incomplete or misspecified.

  • Policy entropy is only a proxy for uncertainty and can confuse genuine uncertainty with action multimodality.It does not directly estimate whether teacher guidance will be helpful in a particular state.
  • The experiments focus on discrete action spaces, while current VLMs perform poorly on precise low-level continuous actions such as torques or velocities.The paper suggests high-level subgoals, skills, or action abstractions as more realistic extensions.
  • SAGE assumes the teacher retains task-relevant competence; uninformative or systematically misleading guidance can produce negative results, without guaranteed robustness.The authors identify estimating teacher usefulness before executing or distilling interventions as future work.
  • ALFWorld is a preliminary higher-dimensional household-interaction stress test using a smaller training budget than the controlled environments.The controlled environments target sparse-reward exploration rather than the full complexity of real-world embodied interaction.
  • SAGE may distill systematic biases or unsafe behaviors from flawed teacher guidance, and advantage weighting does not guarantee safety or alignment under misspecified rewards.The reported ablation also found no consistent robustness benefit over plain BC.
  • The experiments rely on publicly available environments and models whose use is subject to their respective licenses and model terms.The authors state that code, prompts, and CardMaze assets will be released under a permissive license with third-party requirements documented.

A Experiment Hyperparameters

The appendix reports the hyperparameters used for SAGE and PPO training across all environments.

  • Tables 5 and 6 report the hyperparameters used for SAGE and PPO training, respectively, across all environments.

B Hyperparameter Sweep

The CardMaze sweep examines the guidance threshold and objective-weight parameter across multiple seeds, indicating useful ranges for both settings.

  • With ν = 0.25 fixed, β performs best around 1, while ν works well across roughly 0.05–0.45 when β = 1.The sweeps use three seeds per value on CardMaze.

C VLM Query Budget over Training

SAGE concentrates VLM queries early in training and reduces them as policy entropy falls, while using far fewer calls than full-query baselines and none at evaluation.

  • SAGE logs VLM calls during training and reports total calls plus query rate across environments.The query rate is the fraction of environment steps on which the teacher was prompted.
  • More than half of SAGE’s query budget is typically spent before training midpoint, with curves flattening as the entropy gate closes.EZPoints plateaus almost immediately; CardMaze and ALFWorld request guidance longer because of uncertainty and exploration.
  • Full-query VLM-as-policy and LVLM2P issue 100,000 calls at 100k steps, or 7.5× to 86× more than SAGE.DAgger issues 25,000 calls during shortened 25k-step runs, still exceeding SAGE on every controlled environment.
  • SAGE policies act autonomously and issue no VLM calls at evaluation time.This contrasts with VLM-as-policy, which continues paying one call per action after training.

D Qualitative Analysis and Failure Cases of VLM Guidance

Qualitative examples show that VLM guidance can fail on precise spatial reasoning and probability-distribution generation, while SAGE simplifies the required output to an action.

  • The reported examples cover failures from both SAGE and LVLM2P involving non-optimal or poorly specified VLM guidance.
  • SAGE failure cases include VLMs misunderstanding grid worlds and misjudging spatial relationships.The authors suggest image processing or limited pixel-art grid-world prevalence in pretraining as possible reasons.
  • LVLM2P-style prompting struggles to produce exact action distributions as the number of actions grows.The example highlights a mismatch between reasoning and probabilities, including probability assigned to an invalid action.
  • SAGE mitigates distribution-generation difficulty by asking the VLM only for the optimal action.This avoids requiring the teacher to specify a full probability distribution over actions.

E Computational Resources

The experiments used NVIDIA A100 GPUs, with resource allocation varying by VLM size and an estimated total of around 4000 GPU hours.

  • Experiments ran on a cluster equipped with NVIDIA A100 GPUs with 64GB of VRAM.
  • Runs with Qwen3.5-27B or Gemma3-27B used two GPUs, while other runs used one.
  • The authors estimate approximately 4000 GPU hours across preliminary experiments and reported runs.

F VLM Guidance Prompts

The guidance prompts specify visual observations, legal discrete actions, reasoning and action-output formats, and task-specific examples for navigation and arithmetic environments.

  • SAGE prompts replace runtime placeholders with the current state and corresponding image, mission, direction, and admissible-command information.
  • The prompt appendix documents environment-specific hyperparameters and a distribution-generation failure relevant to LVLM2P.The referenced tables cover SAGE and PPO settings, while the failure assigns probability to an invalid action.
  • FrozenLake prompts describe pixel-art cells, hazards, start and goal markers, the elf agent, and four movement actions.The action indices map to left, down, right, and up.
  • Prompt examples demonstrate selecting safe progress in FrozenLake and building 12 from card values through sequential legal actions.One example chooses downward movement; another constructs 2 + 10 = 12.
  • EZPoints prompts require constructing a formula equal to 12 using card numbers once, with legal actions for numbers and arithmetic operators.Face cards count as 10 and aces as 1.
  • The output format requires chain-of-thought inside <think> tags and the selected action index inside <action> tags.Text outside those tags is ignored.
Loading 2609.01567v2…