Source-linked AI summary
The Next Screenshot Knows: Gated Hindsight Distillation for Mobile GUI Agents
Weiwei Li, Junzhuo Liu, Tong Chu, Hengfu Yu, Wen Li
TL;DR
Offline GUI-agent training often discards the future-screen evidence that explains why an action is correct. GHD uses that next screenshot as privileged training information to distill hindsight-grounded knowledge into a prefix-only student, consistently improving task success on AndroidWorld and AndroidLab over standard baselines.
Problem
Offline GUI-agent training predicts actions from observable prefixes even though the evidence justifying many actions often appears only in future states.
Method
Gated Hindsight Distillation uses a future-aware teacher and gated corrections to transfer next-screenshot evidence into a prefix-only student policy.
Results
GHD consistently improves task success over SFT and GRPO on AndroidWorld and AndroidLab across different vision-language models.
Takeaways & Limitations
GHD provides hindsight-grounded knowledge to offline GUI agents without requiring future states or privileged information at deployment.
Takeaways & Limitations
The reported training-time comparison with dynamic sampling is indicative rather than controlled because experiments used non-dedicated rented instances.
Abstract
from arXiv · showhide
GUI agents are commonly trained offline from successful interaction trajectories. Standard training decomposes each trajectory into prefix-action pairs: the agent predicts an action from the current screen and interaction history, while the subsequent observation is discarded. This removes the rationale of why an action is correct: the evidence often appears only on the subsequent screen. For example, to enable Soft Wrap, the agent should click Edit or View, but nothing reveals this until the menu opens. Without such evidence, standard imitation gives the model little chance of ever sampling and thus learning the correct reasoning. To address this issue, we propose Gated Hindsight Distillation (GHD), which uses the next screenshot as privileged information during training. A student predicts from the observable trajectory prefix, while a parameter-sharing teacher additionally observes the next screenshot and re-scores the student's on-policy responses. We apply distillation only when the student fails and the hindsight-conditioned teacher recovers the demonstrated action. GHD improves task success over GRPO on AndroidWorld and AndroidLab across two vision-language models. The code and checkpoints will be made available.
Introduction
Standard prefix-only imitation learning asks GUI agents to reproduce actions without supervising why they are appropriate, even though the evidence often appears only in the next screenshot. Gated Hindsight Distillation uses that future observation as privileged training information, selectively distilling verified corrections into a prefix-only policy and improving benchmark success.
- Motivation: GUI-agent imitation learning supervises demonstrated actions from trajectory prefixes, but later observations often contain the evidence needed to learn grounded action reasoning.The next screenshot can reveal both what an action did and why it was useful, while standard training discards that information.
- Method: Gated Hindsight Distillation conditions a parameter-sharing teacher on the prefix and next screenshot, then distills its future-grounded rationale into a student observing only the prefix.The next observation identifies the demonstrated action’s outcome and explains how it serves the task.
- Method: Distillation applies only when the prefix-conditioned student fails and the hindsight-conditioned teacher predicts the demonstrated action, filtering ambiguous or unreliable targets.This hindsight-correction gate concentrates supervision on cases where the next screenshot provides a verifiable correction.
- Results: GHD consistently improves task success over Supervised Fine-Tuning and Group Relative Policy Optimization on AndroidWorld and AndroidLab.Its largest gains occur for actions requiring application-specific navigation knowledge or implicit prerequisites.
Related work
Related work spans offline behavior-cloning and reinforcement-learning training for GUI agents, future-state use through prediction or verification, and on-policy distillation methods. GHD is positioned within this literature as transferring hindsight from a more informed teacher.
- Offline GUI Agent Training: GUI agents are typically trained offline by behavior cloning successful trajectories with supervised fine-tuning, often followed by GRPO-family reinforcement learning.The cited work includes foundation models specialized for screen understanding and element grounding.
- Leveraging Future for GUI Agents Training: GUI world models predict post-action UI states for lookahead planning, while action-effect verification uses realized next screens to detect and recover from failed actions.Both approaches consume future states online at test time.
- On-Policy Distillation for GUI Agents: GUI-agent distillation transfers grounding and click quality, reinforces via self-distillation, or fills off-trajectory supervision gaps with generated continuations.GHD’s distinguishing axis is that its teacher is strictly more informed.
Method
Gated Hindsight Distillation (GHD) uses the demonstrated next screenshot as training-only privileged supervision for a prefix-conditioned GUI policy. A parameter-sharing teacher supplies token-level corrections only when the student fails and the teacher verifiably recovers the demonstrated action, while inference remains student-only.
- Core mechanism: GHD gives a parameter-sharing teacher the next screenshot to re-score student responses while the student observes only the interaction prefix.The teacher’s privileged context is the current prefix plus the realized next screenshot; the student retains causal deployment inputs.
- Distillation objective: The teacher is not separately trained or autoregressively decoded; it evaluates the student’s own possibly imperfect prefixes to provide dense corrections over reasoning and tool-call tokens.Gradients flow only through the student distribution, and the loss evaluates the student’s top-K tokens plus a residual vocabulary bucket with K = 100.
- Gating: The gate retains distillation only when the prefix-only student fails and the hindsight-conditioned teacher recovers the demonstrated action.Teacher recovery is tested by taking position-wise top-1 predictions along the student response and parsing the resulting action.
- Gating: Teacher recovery requires matching valid JSON tool calls, action names, coordinates within δ = 20, and task-specific text or discrete-argument criteria.Malformed outputs, missing fields, and invalid values are rejected.
- Optimization and inference: GHD complements GRPO: reinforcement learning uses verifier rewards for every rollout, while GHD adds future-grounded token supervision only for verifiably corrected failures.At inference, only the prefix-conditioned policy is retained.
Experiments
Experiments evaluate GHD on AndroidWorld and AndroidLab using controlled comparisons, ablations, privileged-information studies, efficiency analyses, and application-level breakdowns. Results consistently attribute the strongest gains to conditioning the teacher on future observations and transferring targeted supervision to the student.
- Overall comparison: GHD achieves the best Pass@1 among open-data methods at both 7B and 8B scales across AndroidWorld and AndroidLab.Pass@1 is task-level success with one rollout; Pass@3 is success in at least one of three independent rollouts.
- Ablation: 3.17 points are added by introducing the next screenshot, bringing full GHD to 52.73 Pass@1, the largest incremental ablation gain.Gating first contributes 0.71 points over GRPO, while dynamic sampling increases the gain to 2.43 points.
- Privileged information: Adding the next observation raises Pass@1 from 60.34 to 64.67 after reasoning and action are provided, substantially exceeding the 1.29-point gain from the reference answer alone.Table 4 keeps the complete GHD pipeline fixed and varies only the teacher’s privileged information.
- Efficiency and sampling overhead: 2.29 points are gained on AndroidWorld and 14.50 points on AndroidLab without dynamic sampling, while dynamic sampling adds 2.83 and 2.18 points respectively.Dynamic sampling uses at most three attempts and averages 2.69 attempts per batch; observed training time did not materially increase, though this wallclock comparison is indicative rather than controlled.
- AndroidLab applications: GHD outperforms GRPO on seven of nine AndroidLab applications, with the largest gains on Bluecoins and Contacts.Calendar and Zoom are the exceptions, and Zoom contains only five tasks.
Conclusion
The conclusion identifies a supervision gap in offline GUI-agent training: action-justifying evidence often appears only in future states. It presents Gated Hindsight Distillation (GHD) as a future-aware teacher framework that transfers hindsight-grounded knowledge without requiring future states at deployment.
- Supervision gap: Offline GUI agents often lack the evidence that justifies an action because it appears only in future states.Models predict from past and current observations, while the relevant rationale may emerge only afterward.
- Gated Hindsight Distillation: GHD uses future information as a privileged training signal through a future-aware teacher model.The framework extracts application-specific knowledge for distillation during training.
- Deployment: GHD equips offline GUI agents with hindsight-grounded knowledge without requiring future states or privileged information at deployment.This enables deployment without access to the future information used during training.