Source-linked AI summary
Selective Knowledge Control for Continual GUI Agent Learning over Application Streams
Zirui Shang, Xin Shu, Yang Liu, Zhi Gao, Xinxiao Wu, Lifeng Fan
TL;DR
GUI agents must learn new applications while retaining prior knowledge, but application streams mix reusable capabilities with application-specific behaviors. The paper introduces activation-conditioned selective knowledge control, which manipulates neuron-level gradients according to activation, and reports stronger retention alongside robust adaptation on a multi-app sequential benchmark.
Problem
GUI agents must adapt to new applications while retaining prior knowledge, yet new applications combine shared capabilities with distinct workflows that require selective knowledge protection.
Source: §1 · p. 1Method
Activation-conditioned selective knowledge control uses neuron-level gradient manipulation to protect inactive historical neurons and constrain updates to activated ones while allowing compatible adaptation.
Source: §1 · pp. 1–2Results
The method improves overall success rate at every comparable stage from Stage 2 to Stage 8, with increases of 12.6, 7.1, and 10.6 percentage points at Stages 3, 6, and 7, respectively.
Source: §4.2 · p. 5Takeaways & Limitations
Abstract
from arXiv · showhide
Continual learning is a crucial capability for Graphical User Interface (GUI) agents to adapt to evolving applications while retaining knowledge acquired from previous applications. Such application streams pose a challenging knowledge modeling problem: new applications often share underlying knowledge with past ones, yet also introduce specific knowledge that must not interfere with historical knowledge. In this paper, we propose activation-conditioned selective knowledge control, a lightweight method that achieves selective knowledge retention via neuron-level gradient manipulation. Our method maintains a compact historical knowledge state to protect highly activated MLP neurons that preserve previous knowledge. When a new application arrives, it performs real-time gradient surgery conditioned on forward activation. Concretely, the protected neurons are categorized into two types: unactivated neurons holding specific knowledge, whose gradients are truncated to prevent interference; and activated neurons holding shared knowledge, whose gradients are orthogonally projected to preserve stability while enabling adaptation. After each application stage, newly identified critical neurons are merged into the historical state for future learning. Empirical evaluations on multi-app sequential benchmark demonstrate that our method effectively mitigates catastrophic forgetting on prior applications while sustaining robust adaptation to new ones.
1 Introduction
GUI agents must adapt to evolving applications while retaining prior knowledge, but new applications combine reusable operations with application-specific workflows. The paper proposes activation-conditioned selective knowledge control to protect historical knowledge selectively while keeping shared knowledge adaptable.
1 Introduction
- Continual GUI learning must retain knowledge from prior applications while adapting to new ones that combine shared operations with application-specific workflows.The paper distinguishes this selective knowledge problem from the conventional trade-off between forgetting and fast adaptation.Source: §1 · p. 1
- Existing preservation methods treat historical knowledge uniformly, which can constrain shared updates without distinguishing reusable knowledge from knowledge that should be protected.The paper identifies this limitation in parameter freezing, importance regularization, gradient surgery, and fixed neuron-level protection.Source: §1 · p. 1
- The paper proposes activation-conditioned selective knowledge control to distinguish protected application-specific knowledge from shared knowledge that should remain adaptable.It frames knowledge control as a dynamic distinction rather than a static preservation rule.Source: §1 · pp. 1–2
- Neuron-level gradient surgery truncates gradients for inactive protected neurons and projects gradients for activated protected neurons orthogonally to historical update directions.After each application stage, newly identified important neurons are merged into the historical state; the procedure needs no old-application replay, gradient computation, or full application-specific checkpoints.Source: §1 · p. 2
- Empirical evaluations on a multi-app sequential benchmark show reduced performance drops on previous applications alongside robust adaptation to new applications.The paper reports this outcome as evidence that the method mitigates catastrophic forgetting while sustaining adaptation.Source: §1 · p. 2
2 Related Work
The related work covers GUI-agent architectures and training, then reviews continual-learning approaches and recent work on GUI-agent adaptation. It positions this method as sequential gradient control of one model rather than post-hoc fusion of separately trained models.
2.1 GUI Agents
- GUI agents span structured, visual, and hybrid systems, while training has shifted from supervised fine-tuning toward reinforcement fine-tuning and benchmarks cover increasingly varied tasks.The paper notes that most training pipelines still assume a fixed data distribution despite changing applications, layouts, workflows, and tool APIs.Source: §2.1 · p. 2
2.2 Continual Learning
- Prior continual-learning work includes preservation constraints, replay, and gradient-space methods, while recent GUI-agent work targets adaptation without directly controlling new updates against historical GUI knowledge.Unlike Agent-Dice’s post-hoc fusion of separately trained models, this paper controls gradients during sequential fine-tuning of a single model.Source: §2.2 · p. 3
3 Method
The method distinguishes shared from application-specific historical knowledge, then uses current activations to condition neuron-level gradient updates. Between application stages, it expands its protected-neuron state and stores compact historical direction subspaces.
3.1 Problem Setup and Formulation
- The formulation separates shared knowledge, whose updates should remain orthogonal to historical loss gradients, from application-specific knowledge, whose updates are set to zero.Source: §3.1 · p. 3
3.2 Activation-Conditioned Forward Partitioning
- At each training stage, the method selects the top ⌊ρdff⌋ activated neurons and partitions protected historical neurons by whether the current update reuses them.The method assigns inactive protected neurons to application-specific regions and activated protected neurons to shared functional regions.Source: §3.2 · pp. 3–4
3.3 Neuron-Level Backward Gradient Surgery
- For protected neurons, it blocks updates when they are inactive and projects shared-neuron gradients onto the orthogonal complement of historical directions, preserving those directions while allowing adaptation.Source: §3.3 · p. 4
3.4 Stage-End State Update
- After each stage, the method expands the protected set with newly identified important neurons.Source: §3.4 · pp. 4–5
- For each protected neuron, truncated SVD retains a compact orthonormal basis of dominant historical parameter-displacement directions, capped at rsvd directions.Source: §3.4 · p. 5
- The stage-end update runs between stages; online training requires neither historical trajectory replay nor old-application gradients or full application-specific checkpoints.Source: §3.4 · p. 5
4 Experiments
The experiments evaluate selective knowledge control in an eight-application OSWorld stream, comparing its success rates, retention–adaptation balance, ablations, hyperparameters, overhead, and a qualitative transfer case. The reported results show stronger stream-wide performance and retention while maintaining adaptation to new applications.
4.1 Experimental Setup
- The evaluation uses an eight-stage OSWorld application stream and tests each stage on all applications seen so far; RP, CP, and RAH distinguish retention from current-stage adaptation.The stream runs from VLC through Thunderbird, LibreOffice Writer, Impress and Calc, GIMP, VSCode, and Chrome.Source: §4.1 · p. 5
4.2 Main Results
- Overall success rate is higher than the baseline at every comparable stage from 2–8, with gains of 12.6, 7.1, and 10.6 percentage points at Stages 3, 6, and 7.Table 1 reports average success rates over seen applications after each stage.Source: §4.2 · p. 5, Table 1 · p. 6
- RP, CP, and RAH improve by 6.3, 9.1, and 9.8 percentage points on average, respectively, indicating gains in retention and current-application adaptation.RP measures historical retention, CP measures success on the newly introduced application, and RAH is their harmonic mean.Source: §4.2 · p. 5, §4.1 · p. 5, Table 2 · p. 6
- Replay preserves prior performance to some extent, but the proposed method is more effective at balancing adaptation and retention against replay, EWC, and LoRA.The comparison reports stage-wise success rates and overall performance under the same training configuration.Source: §4.2 · pp. 5–6, Table 3 · p. 6
4.3 Ablation Studies
- Activation-aware selective control reaches a 47.1% mean success rate, versus 41.5% for static projection, 40.1% for static freezing, and 37.6% for the baseline.The method also achieves the best performance on Stage 2 (43.8%) and Stage 3 (50.3%).Source: §4.3 · pp. 6–7, Table 4 · p. 7
- ρ = 2% gives 50.3% overall success across three observed applications, while rsvd = 4 gives 42.7% overall success at Stage 8.The reported sweeps favor a moderate selection ratio and a moderate number of retained historical update directions.Source: Table 5 · p. 7, Table 6 · p. 7, §4.3 · p. 7
4.4 Additional Analysis
- 1.0% per-step time increase—from 4279.23s to 4324.00s—accompanies 3.3% lower token throughput and 1.9% lower compute throughput.The overhead comparison uses the same training configuration.Source: Table 7 · p. 7, §4.4 · p. 7
- The Stage 7 VSCode installation workflow transfers to a Stage 8 Chrome extension task, while the baseline fails to confirm the source or verify installation.The case describes reuse of the general procedure through Chrome-specific controls.Source: §4.4 · p. 7
5 Conclusion
The paper concludes that activation-conditioned selective knowledge control provides neuron-level control of historical knowledge for continual GUI-agent learning. Its multi-app evaluation reports reduced performance drops on previous applications while sustaining robust adaptation to new ones.
5 Conclusion
- Activation-conditioned selective knowledge control uses selective, neuron-level gradient-based control over historical knowledge rather than uniform knowledge management.The paper presents it as a method for continual learning over application streams.Source: §5 · p. 7
- The approach preserves application-specific knowledge while strengthening shared knowledge across applications.Figure 3 is identified as a case of shared procedural knowledge reuse on a Chrome task.Source: §5 · p. 7, Figure 3 · p. 8
- Multi-app sequential benchmark evaluations show mitigated performance drops on previous applications while sustaining robust adaptation to new applications.Source: §5 · p. 7
6 Supplementary Material
The supplementary evaluations report that the method outperforms the baseline under a shuffled stream and illustrate procedural knowledge reused across applications. Layer-wise offsets suggest it redirects updates rather than merely shrinking them.
6.1 Training Order Comparison
- Under the shuffled VLC → VS Code → GIMP order, the proposed method outperforms the baseline across stages on overall performance.Table 8 reports stage-wise success rates and overall performance for both methods under this shuffled order.Source: Table 8 · p. 11, §6.1 · p. 11
6.2 Additional Case Studies
- Writer-to-Impress transfer shows that selecting the correct content scope before formatting distinguishes task success from the baseline’s local-span failure.The baseline changes only a local span, leaving mixed font sizes and failing the task.Source: §6.2 · p. 11
- VLC-to-Chrome transfer succeeds when the method verifies the requested destination state before submission, unlike the baseline.The method checks and changes Chrome’s bookmark folder to Bookmarks bar before clicking Done; the baseline clicks Done without changing or verifying it.Source: §6.2 · p. 11, Figure 5 · p. 13
6.3 Parameter-Offset Heatmap
- Across 196 layer–stage pairs, mean direction similarity is low; median displacement ratio is 1.088, and only 37% have Rℓ,t < 1.Stage 4 offsets consistently oppose the baseline’s, while Stage 5 offsets are mostly orthogonal and 3.5–4.2 times as large across layers; overall, updates are redirected rather than merely suppressed.Source: §6.3 · p. 11, Figure 6 · p. 14