Source-linked AI summary
ComputerSD: Online Self-Distillation from Real-Time Feedback for Computer-Use Agents
Yong Du, Tongbo Chen, Zhengxi Lu, Yizhou Liu, Bofan Chen, Tao Jiang, Wenhao Xu, Yongliang Shen
TL;DR
Sparse outcome rewards leave intermediate actions unsupervised, and applying OPSD to online CUA training raises guidance-misalignment and signal-reliability challenges. ComputerSD uses post-action GUI feedback to guide and gate token-level OPSD alongside trajectory-level GRPO in asynchronous training. It outperforms outcome-only GRPO on OSWorld-Verified and improves performance in out-of-distribution settings.
Problem
Outcome rewards indicate task completion but do not identify which intermediate actions were productive, redundant, or erroneous, while fixed OPSD guidance can misalign with the student’s state and induce signals that conflict with step correctness.
Source: §1 · p. 2Method
ComputerSD uses a fine-tuned GUI analyzer to produce post-action guidance and step-level value judgments that gate token-level OPSD, jointly optimized with trajectory-level GRPO in an asynchronous framework.
Source: §1 · pp. 2–3Results
Takeaways & Limitations
ComputerSD improves in-domain and out-of-distribution performance on both backbones, including held-out OSWorld-Verified categories and WindowsAgentArena.
Source: §4.2 · p. 7ComputerSD depends on reliable analyzer guidance and value judgments, and the value gate cannot fully resolve errors when both outputs are unreliable.
Source: A LIMITATIONS · p. 13
Abstract
from arXiv · showhide
Online training enables computer-use agents (CUAs) to improve through interaction with executable environments. However, existing methods primarily rely on sparse outcome rewards, which provide no supervision for intermediate actions. On-policy self-distillation (OPSD) offers token-level learning signals through privileged rescoring, but directly applying it to CUA online training presents two challenges: fixed guidance may become misaligned with the student's current state, and guidance-induced probability shifts may conflict with step-level correctness. We introduce ComputerSD, an online self-distillation method for CUAs that converts real-time feedback from executed GUI transitions into guidance for policy learning. A fine-tuned GUI analyzer produces guidance and a step-level value score after each action; the guidance provides privileged context, while the score regulates the resulting OPSD signals. ComputerSD jointly optimizes token-level OPSD and trajectory-level GRPO in a fully asynchronous training framework. On OSWorld-Verified, ComputerSD outperforms outcome-only GRPO by 1.9 and 4.1 percentage points on the general-purpose Qwen3-VL-8B-Thinking and specialized EvoCUA-8B backbones, respectively. Evaluation in out-of-distribution settings further supports the generalizability of ComputerSD. These results demonstrate the effectiveness of learning from real-time feedback through online self-distillation for CUAs.
1 INTRODUCTION
ComputerSD addresses sparse outcome rewards and the misalignment and unreliability of token-level OPSD signals by using feedback from executed GUI transitions to guide and regulate online learning. On OSWorld-Verified, it outperforms outcome-only GRPO on two backbones.
1 INTRODUCTION
- Outcome rewards indicate task completion but not which episode actions were productive, redundant, or erroneous, leaving intermediate-action credit assignment unresolved.OPSD can supply token-level supervision through privileged rescoring of the policy’s own sampled responses.Source: §1 · p. 2
- Fixed guidance can become misaligned when the student leaves an assumed path, while guidance-induced token shifts can conflict with step correctness.Across training, over 80% of tokens at correct steps are suppressed and nearly 20% at incorrect steps are reinforced.Source: §1 · p. 2
- Post-action feedback describes the state reached and judges step correctness, addressing guidance misalignment while regulating the token-level signals it induces.The paper distinguishes this value judgment from prior approaches that stop at matching guidance to the post-action state.Source: §1 · p. 2
- A fine-tuned GUI analyzer supplies guidance and a step-level value score after each action; a value gate weights token-level shifts, jointly optimized with trajectory-level GRPO.The asynchronous pipeline is designed to prevent per-step analysis from stalling rollout.Source: §1 · p. 2
- ComputerSD raises OSWorld-Verified success from 37.9% to 39.8% for Qwen3-VL-8B-Thinking and from 43.8% to 47.9% for EvoCUA-8B, versus outcome-only GRPO.Removing real-time feedback or the value gate lowers performance below GRPO.Source: §1 · pp. 2–3
- The fully asynchronous pipeline overlaps environment interaction, GUI analysis, privileged rescoring, and policy optimization while rollout workers continue collecting trajectories.This pipeline is presented as a contribution alongside the value-gated self-distillation method.Source: §1 · p. 3
2 RELATED WORK
Prior online CUA methods use outcome rewards or scalar step-level signals, while OPSD methods use privileged information to provide token-level supervision. ComputerSD combines GUI-step feedback with token-level supervision and overlaps analysis with rollout and training.
2 RELATED WORK
- Online CUA methods use environment rewards or scalar step-level estimates, whereas ComputerSD turns feedback on each executed step into token-level supervision and overlaps analysis with rollout and training.Related approaches include GUI-Shepherd’s process reward model and GiGPO’s repeated-state step advantages (Feng et al., 2025).Source: §2 · p. 3
- Prior multi-turn OPSD uses trajectory-derived privileged information, while GUI methods condition on post-action observations or hints; ComputerSD gates each token signal using a fine-tuned analyzer’s step-level judgment.This differs from gating by teacher–student gap or trajectory outcome.Source: §2 · p. 3
3 METHOD
ComputerSD trains CUAs by pairing online trajectory-level GRPO with analyzer-derived, value-gated token-level self-distillation.
3 METHOD
- ComputerSD combines token-level OPSD with trajectory-level GRPO to optimize the policy using real-time GUI feedback.The analyzer supplies step-level guidance and value scores; guidance provides privileged context for rescoring.Source: §3 · p. 3, §3.3 · p. 5
3.1 PROBLEM FORMULATION
- CUA execution is formulated as a partially observable process in which the policy samples reasoning and executable actions from task and observation context.Executing an action yields the next observation; each trajectory ends with an environment-verifier reward.Source: §3.1 · p. 3
3.2 GUI ANALYZER SUPERVISED FINE-TUNING
- The GUI analyzer is trained on expert annotations of step contexts, value scores, and guidance, then frozen for online policy training.Collected trajectories provide step contexts; the analyzer is initialized from the base policy and fine-tuned to predict the annotations.Source: §3.2 · p. 4
3.3 VALUE-GATED ON-POLICY SELF-DISTILLATION
- A value gate weights token-level log-probability shifts by step-level value judgments, reinforcing aligned signals and suppressing misaligned ones.The resulting value-gated OPSD objective is combined with trajectory-level GRPO.Source: §3.3 · p. 5
- Training asynchronously overlaps environment interaction, GUI analysis, privileged rescoring, and policy optimization.The trainer updates after collecting a trajectory batch and publishes updated parameters to rollout workers.Source: §3.3 · p. 5
4 EXPERIMENTS
ComputerSD outperforms outcome-only GRPO on OSWorld-Verified across both backbones and improves results in held-out and cross-platform settings. Training-dynamics and ablation analyses examine efficiency and the contributions of feedback, value gating, and OPSD design.
4.2 MAIN RESULTS
- ComputerSD reaches 39.8% on Qwen3-VL-8B-Thinking, exceeding GRPO by 1.9 points, and reaches 47.9% on EvoCUA-8B, exceeding GRPO by 4.1 points.Table 1 reports success rate as Pass@1.Source: Table 1 · p. 6, §4.2 · pp. 6–7
4.1 EXPERIMENTAL SETTINGS
- Experiments use two 8B backbones, train for 180 updates on OSWorld-Verified, and evaluate on 222 in-domain and 139 OOD tasks plus WindowsAgentArena.The held-out OSWorld-Verified categories are Multiple Apps and Chrome.Source: §4.1 · p. 6
4.2 MAIN RESULTS
- On held-out categories and WindowsAgentArena, ComputerSD beats GRPO by 5.9 and 1.2 points for Qwen3-VL-8B-Thinking, and 5.2 and 3.6 points for EvoCUA-8B, respectively.GRPO improves in-domain performance but degrades most OOD performance relative to the base models on both backbones.Source: Table 2 · p. 7, §4.2 · p. 7
4.3 TRAINING DYNAMICS.
- ComputerSD reaches GRPO’s final performance after about 100 Qwen and 40 Evo updates, using about 56% and 22% of GRPO’s trajectory budget, respectively.It maintains higher success rates than GRPO through most of training on both backbones.Source: Figure 4 · p. 7, §4.3 · p. 7
- Asynchronous ComputerSD stabilizes at about 210 trajectories per hour versus GRPO’s 370, retaining 57% of GRPO throughput; overlap yields 5× synchronous ComputerSD throughput.Analyzer queries and privileged rescoring account for the additional cost.Source: Figure 5 · p. 7, §4.3 · p. 8
4.5 ANALYSIS
- λOPSD=0.01 performs best; at 0.1, KL loss rises by more than an order of magnitude, while 0.001 provides too little token-level supervision.Both larger and smaller coefficients perform worse than the intermediate setting.Source: Figure 7 · p. 8, §4.5 · p. 9
4.4 ABLATION STUDIES
- Removing GUI-analyzer SFT, replacing real-time feedback with fixed guidance, and removing the value gate lower success by 1.8, 4.8, and 3.2 points, respectively.Fixed guidance can become misaligned with the agent’s state; without the value gate, guidance-induced signals are not uniformly reliable.Source: §4.4 · p. 8, Table 3 · p. 9
4.5 ANALYSIS
- Hard value gating drops overall success by 4.6 points; reverse gating widens the teacher–student gap and lowers entropy, unlike the proposed gate.The proposed gate narrows the gap without the entropy decline.Source: Table 4 · p. 9, §4.5 · p. 9, Figure 8 · p. 9
5 CONCLUSION
ComputerSD turns feedback from GUI transitions into policy updates for computer-use agents. On OSWorld-Verified it beats outcome-only GRPO on both backbones and improves held-out and cross-platform performance.
5 CONCLUSION
- ComputerSD combines real-time GUI feedback, value-gated token-level OPSD, trajectory-level GRPO, and fully asynchronous online training; it beats outcome-only GRPO by 1.9 and 4.1 points on two backbones.The conclusion also reports improvements on held-out categories and cross-platform scenarios for both backbones.Source: §5 · p. 9
A LIMITATIONS
ComputerSD’s guidance and step-level value judgments depend on GUI-analyzer reliability, and unreliable outputs can distort supervision.
A LIMITATIONS
- Unreliable analyzer guidance and value judgments can distort OPSD signals; systematic errors may still bias updates despite the small auxiliary loss coefficient.The value gate cannot fully resolve the issue when both outputs are unreliable.Source: A LIMITATIONS · p. 13
B TRAINING DETAILS
The training details describe analyzer fine-tuning, online optimization, and asynchronous infrastructure. The appendices also report the annotation dataset and expert annotation procedure.
B TRAINING DETAILS
- The GUI analyzer is LoRA-fine-tuned; policy optimization is full-parameter with the visual encoder frozen, using separate rollout and training engines.Rollouts use temperature 1.0 and retain the three most recent historical screenshots in context.Source: B TRAINING DETAILS · p. 13
- The asynchronous framework rescoring uses the same policy snapshot for ordinary and privileged contexts, caches gaps and gate weights, and retains samples from earlier policy versions.Online RL uses one node with 16 PPUs, split as 8 PPUs each for rollout and training engines.Source: B TRAINING DETAILS · p. 13
C.1 TRAINING DATA CONSTRUCTION
- The Qwen3-VL-8B-Thinking collection spans 222 tasks and 1,776 trajectories with 34,188 step samples; 43.4% of trajectories succeed and 40.1% of annotated steps are judged correct.Each task contributes eight sampled trajectories.Source: §C.1 · p. 13
C.2 EXPERT ANNOTATION DETAILS
- Expert annotations assess action consistency and effectiveness from the task, pre-action context and response, executed action, and post-action screenshot, then provide do-and-avoid guidance.Kimi K3 is used as the expert model.Source: §C.2 · p. 14
C.3 ANALYZER QUALITY EVALUATION
The GUI analyzer is evaluated against expert step-level judgments and for valid structured output. Fine-tuning improves both measures on 341 held-out samples.
C.3 ANALYZER QUALITY EVALUATION
- The evaluation uses 341 annotated samples held out for validation and excluded from supervised fine-tuning.The base and fine-tuned Qwen3-VL-8B-Thinking analyzers are compared on expert-judgment agreement and output validity.Source: §C.3 · p. 17
- 85.3% expert agreement and 99.7% valid output rate follow fine-tuning, up from 29.9% and 41.6%, respectively.The results support task-specific fine-tuning for value assessment and structured output generation.Source: §C.3 · p. 17, Table 10 · p. 17
- The fine-tuned analyzer provides more reliable supervision while avoiding expensive expert API calls during online training.Valid output rate counts responses that meet the required format and can be parsed by the extraction rules.Source: §C.3 · p. 17
D ADDITIONAL EXPERIMENTAL DETAILS
The experiments use OSWorld-Verified for online training and in-domain and out-of-distribution evaluation, with WindowsAgentArena for cross-platform evaluation. The appendix specifies the tested backbones, baselines, and system prompt.
D.1 DATASETS
- OSWorld-Verified evaluation uses 222 training and in-domain tasks and 139 Chrome and Multiple Apps tasks held out for OOD evaluation; WindowsAgentArena adds cross-platform testing.WindowsAgentArena contains 154 real-Windows tasks, and policies trained on OSWorld-Verified receive no additional Windows-task training.Source: §D.1 · p. 17
D.2 BASELINES
- The study tests Qwen3-VL-8B-Thinking as a general-purpose backbone and EvoCUA-8B as a computer-use model already post-trained on synthetic interaction experience.Both models are 8B-scale; EvoCUA-8B tests whether ComputerSD adds gains to a computer-use-specialized backbone.Source: §D.2 · pp. 17–18
- Baselines include fixed expert guidance without policy updates, outcome-only GRPO, and GRPO with PRM scores summed into trajectory rewards.GRPO with PRM combines the summed step scores with the environment outcome reward before computing trajectory-level advantages.Source: §D.2 · p. 18
D.3 SYSTEM PROMPT
- Trajectory sampling and evaluation use the official system prompt shown in Figure 10.The appendix includes the prompt as a figure.Source: §D.3 · p. 18
E SUPPLEMENTARY RESULTS
Additional OSWorld-Verified comparisons show that prompt-only guidance yields limited gains and GRPO with PRM remains below ComputerSD. The results distinguish inserting guidance from using it for policy learning.
E SUPPLEMENTARY RESULTS
- 39.8% for ComputerSD exceeds 37.4% for GRPO with PRM on OSWorld-Verified; prompt-only guidance reaches 34.6%, up from 33.8%.The prompt-only method inserts expert-generated trajectory guidance without updating policy parameters.Source: E SUPPLEMENTARY RESULTS · p. 18
- A scalar reward alone cannot fully exploit real-time environment feedback, even when it incorporates step-level value assessments.The GRPO-with-PRM baseline sums analyzer scores across the trajectory and combines them with the outcome reward.Source: E SUPPLEMENTARY RESULTS · p. 18
F CASE STUDIES
The case studies contrast outcome-only GRPO failures with ComputerSD successes on an in-domain task and an OOD task. Their trajectories illustrate differences in state handling and verification behavior.
F CASE STUDIES
- On both the in-domain and OOD tasks, the outcome-only GRPO policy fails while the ComputerSD-trained policy succeeds.Tables 12–15 show the corresponding unsuccessful GRPO and successful ComputerSD rollouts.Source: F CASE STUDIES · p. 18, Table 12 · p. 19, Table 13 · p. 21
- On the in-domain settings task, GRPO changes the interval without enabling AutoRecovery and declares success, while ComputerSD enables the checkbox before changing and confirming the interval.The comparison suggests ComputerSD internalizes task-state understanding rather than merely memorizing action sequences.Source: F CASE STUDIES · pp. 18–19, Table 12 · p. 19, Table 13 · p. 21
- On the OOD profile task, ComputerSD reopens the profile menu to verify the updated name before terminating, unlike GRPO, which terminates immediately.The authors describe this verification behavior as suggesting that effective strategies learned from real-time feedback transfer to unseen scenarios.Source: F CASE STUDIES · p. 19, Table 14 · p. 23, Table 15 · p. 24