TSRL: Trigger-based Supervised Reinforcement Learning for Continual Agent Training

1 Introduction

• Continual learning is challenging because acquiring new skills can degrade performance on previously mastered tasks.The introduction frames this as catastrophic forgetting in dynamic real-world environments.Source: §1 · p. 1
• In an illustrative case study, trigger-based supervision trades off new-task acquisition and prior-task retention, unlike the distinct forgetting and convergence behavior of SFT and RL.The introduction describes RL as exhibiting minimal forgetting, SFT as inducing severe forgetting, and trigger-based supervision as achieving a favorable trade-off.Source: §1 · pp. 1–2, Figure 2 · p. 2
• Theoretical analysis links forgetting to cumulative Fisher-weighted updates and motivates sparse supervision on failure states with on-policy RL as the backbone.The paper derives an upper bound on forgetting and presents sparsity via locality and an on-policy foundation as design principles.Source: §1 · p. 2
• TSRL consistently outperforms baselines and resists forgetting across three agent benchmarks: ALFWorld, WebShop, and Deep Search.The introduction specifies ALFWorld and WebShop with 4B and 7B models, and Deep Search with 32B models.Source: §1 · p. 2
• TSRL combines SFT's sample efficiency with RL's forgetting resistance by activating expert supervision only on hard states where the policy fails.RL remains the optimization backbone, while supervised gradients are injected when the Group Success Rate falls below a predefined threshold.Source: §1 · p. 2

2 Preliminaries

• The agent is modeled as a finite-horizon MDP whose states aggregate interaction history, actions are textual outputs, and transitions capture environmental responses.The MDP specifies state and action spaces, a transition kernel, reward function, and maximum interaction horizon.Source: §2 · p. 3
• The paper defines forgetting as the change in prior-task performance from the pretrained policy to the training policy, while post-training seeks high new-task performance with minimal forgetting.The policies distinguished are a pretrained prior, a target-task expert available through offline demonstrations, and the training policy.Source: §2 · p. 3
• SFT is sample-efficient but risks overwriting prior capabilities, whereas RL optimizes from the pretrained policy and typically needs more interaction data because rewards are sparse and delayed.The paper also notes that SFT followed by RL may achieve target-task performance, but later RL may not fully recover forgetting caused during SFT.Source: §2 · p. 3
• GRPO samples a group of responses for a state and normalizes them into group-relative advantages, without requiring a value network.Its policy update uses a clipped objective analogous to PPO, with optional KL regularization.Source: §2 · p. 3

3 Illustrative Case Study

• TSRL achieves a favorable trade-off between new-task acquisition and prior-task retention, while pure RL forgets little but converges slowly and SFT induces severe forgetting.The comparison covers pure SFT, pure RL, SFT→RL, and TSRL over 300 optimization steps; the trigger rate falls as training proceeds, while SFT→RL applies dense supervision for its first 150 steps.Source: §3 · pp. 3–4
• The study's overlapping Gaussian-mixture tasks model a continual-learning setting that requires acquiring novel behavioral patterns while retaining prior knowledge.The four paradigms compared are pure SFT, pure RL, SFT→RL, and TSRL.Source: §3 · p. 3
• The theoretical bound relates forgetting to cumulative updates along prior-task-sensitive directions, which trigger-based supervision restricts to hard states.The analysis characterizes forgetting using a Fisher Information Matrix and reports strictly smaller cumulative energy under trigger-based supervision.Source: §3 · p. 4

4 Methodology

• TSRL operationalizes the theoretical case for sparse, hard-state supervision as a framework for continual agent training.The preceding analysis argues that restricting expert supervision to hard states yields less forgetting than full SFT.Source: §4 · p. 4

4.1 Trigger-based Supervision on Hard State

• Expert supervision triggers when the group success rate falls below a threshold, and improves in policy capability automatically deactivate it for more states.The threshold δ controls activation; δ = 0 triggers when all sampled generations fail, while δ = 0.5 triggers when fewer than half succeed.Source: §4.1 · p. 4

4.2 Policy Optimization

• RL remains the optimization backbone, while triggered SFT uses expert actions on current-policy states and its weight anneals linearly to zero.Annealing reduces late-stage objective mismatch when the policy develops high-reward reasoning patterns that diverge from the expert.Source: §4.2 · pp. 4–5

4.3 Summary and Design Principles

• TSRL implements sparsity via locality by limiting SFT gradients to triggered hard states, while its constant RL gradient preserves an on-policy optimization foundation.The paper identifies these as the framework’s two core design principles.Source: §4.3 · p. 5

5.1 Experimental Settings

• Experiments cover ALFWorld and WebShop with Qwen2.5-7B, and Deep Search with Qwen2.5-32B, comparing TSRL-GRPO/DAPO against SFT, RL, and SFT→RL.The benchmarks target household decision-making, web navigation, and multi-step information retrieval and synthesis.Source: §5.1 · pp. 5–6

5.2 Main Results

• TSRL GRPO improves over SFT→GRPO by 4.3% on ALFWorld and 3.5% on WebShop success rate; TSRL DAPO gains 3.4% and 4.4%, respectively.Source: Table 2 · p. 6, §5.2 · p. 6
• In two-task transfer, TSRL alone combines strong new-task acquisition with prior-task preservation for WebShop-after-ALFWorld; in reverse, it best balances retention and target performance.The reported trend also holds in a longer three-task sequence.Source: Table 3 · p. 6, §5.2 · p. 7
• At 32B, TSRL achieves performance comparable to SFT→RL on Deep Search while better preserving math reasoning; on AIME 25, it surpasses the base model.Source: Table 4 · p. 7, §5.2 · p. 7

5.3 Analysis

• In Deep Search training dynamics, TSRL's selective expert demonstrations enable faster convergence and higher performance than SFT→RL.The analysis attributes RL's lower sample efficiency to response-level rewards, which make credit assignment harder than dense token-level SFT supervision.Source: §5.3 · p. 7, Figure 3 · p. 7
• On ALFWorld and WebShop, triggering SFT only when the entire group fails (δ = 0) uses expert data most effectively; higher thresholds hurt performance.The authors interpret this as evidence that uniform SFT can impair subsequent RL performance.Source: Figure 4 · p. 8, §5.3 · p. 8
• Fixed SFT-loss weights of 1.0 and 0.1 degrade performance, while annealing over 50 steps performs comparably to TSRL's 100-step schedule.The results indicate reasonable robustness to annealing duration, but persistent supervision adversely affects final performance.Source: §5.3 · p. 8

Reinforcement Learning for LLM Agents

• Recent agent-RL work mainly improves credit assignment, training stability, and uncertainty handling, whereas this paper focuses on incorporating expert data without erasing prior capabilities.Source: §6 · p. 8

Catastrophic Forgetting in Continual Training

• Prior evidence reports less forgetting with RL than SFT, while hybrid methods such as UFT and LUFFY mainly target single-task reasoning rather than continual agent training.Source: §6 · p. 8
• TSRL is distinguished by using sparse, outcome-based supervision specifically to control forgetting in continual agent training.Source: §6 · p. 8

• TSRL combines SFT's sample efficiency with RL's forgetting resistance for continual agent training.Source: §7 · p. 8
• The method keeps RL as its optimization backbone and activates expert supervision only on hard states where the policy fails.This makes expert guidance sparse and localized to difficult tasks.Source: §7 · p. 8
• Across three agent benchmarks, TSRL outperforms baselines and retains more capability on prior tasks in continual learning scenarios.Source: §7 · p. 8

Limitations

• Expert-trajectory quality and binary GSR constrain TSRL: noisy or suboptimal demonstrations may weaken supervision, and dense or continuous rewards may make the trigger less informative about state difficulty.The experiments retain only maximum-reward trajectories, a filtering step that may be impractical in real-world settings.Source: Limitations · p. 9

A Pseudocode of TSRL

• Each training iteration samples states, computes group success rates and RL advantages, adds expert SFT loss when GSR is at or below δ, then updates the policy.The procedure supports online expert queries or retrieval of offline demonstrations.Source: A Pseudocode of TSRL · p. 12

B.1 Experimental Setup

• The illustrative case models prior and new tasks with bimodal Gaussian mixtures, with the new-task distribution sharing a prior mode and adding a novel mode.This configuration represents partial overlap with the prior task alongside acquisition of new behavioral patterns.Source: §B.1 · p. 12
• Four paradigms are compared over 300 optimization steps: pure SFT, pure RL, SFT→RL, and TSRL, with prior-task retention and new-task acquisition measured by negative KL divergence.TSRL triggers SFT when GSR falls below threshold δ.Source: §B.1 · p. 12

B.2 Extended Analysis

• As training progresses, the trigger deactivates on more queries, shifting TSRL from early expert guidance on failures toward RL exploration as competence develops.The declining trigger frequency is shown in Figure 2(c).Source: §B.2 · p. 13
• Dense SFT in SFT→RL causes substantial drift from the base model that later RL only partially reverses, whereas TSRL interleaves RL with SFT limited to difficult instances.The authors associate the dense phase with overwriting parameters important for prior-task performance.Source: §B.2 · p. 13
• Restricting supervision to hard states keeps cumulative supervised-update energy small relative to full SFT, aligning the empirical pattern with the Fisher-weighted forgetting bound.The passage describes sparse supervision as a practical instantiation of limiting destructive parameter drift.Source: §B.2 · p. 13

C.1 Fisher Information Characterization of Forgetting

• A local quadratic approximation uses the prior-task Fisher information matrix to link parameter drift from the pretrained policy to functional degradation on prior data.The authors restrict this approximation to the neighborhood of the pretrained prior policy rather than assuming a globally quadratic loss landscape.Source: §C.1 · pp. 13–14
• Forgetting is associated with Fisher-weighted displacement, so the training trajectory should avoid directions with high curvature on the prior-task manifold.The analysis evaluates parameter changes using the geometry induced by the prior task.Source: §C.1 · p. 14

C.2 Forgetting Reduction via Selective Supervision

• Under the stated assumptions, the forgetting bound depends on cumulative update energy, and selective supervision has lower energy than full SFT when the policy does not fail universally.The trigger aggregates supervised gradients only from failing states, unlike full SFT, which aggregates across the state distribution.Source: §C.2 · p. 14

C.3 Design Principles

• The design principles are to trigger supervision locally on hard states and retain on-policy RL as the primary optimization signal.The paper attributes reduced supervised-update energy to locality and contrasts current-policy learning with offline SFT toward an external distribution.Source: §C.3 · p. 14

D Extended Related Work

• Related RL-agent work addresses credit assignment, entropy management, and training stability, while prior analyses attribute RL’s lower forgetting to reward-scaled or base-proximal updates and mode-seeking behavior.The cited work also describes hybrid strategies including parameter freezing, rollout filtering, cascade training, and iterative SFT-style updates.Source: §D.1 · p. 15, §D.2.1 · p. 15
• Unlike UFT’s dense supervision and LUFFY’s off-policy trajectory mixing, TSRL uses a state-dependent trigger; the paper positions it as avoiding auxiliary models, second-order statistics, and replay mechanisms used by classical continual-learning approaches.Table A1 summarizes positioning among hybrid SFT-RL methods.Source: §D.2.2 · p. 16, §D.2.3 · p. 16, Table A1 · p. 16

E Additional Continual Learning Experiments

• Additional experiments are framed to test whether TSRL’s benefits persist across longer task chains, stronger hybrid baselines, and a different model.The three-stage experiment evaluates final-task performance alongside retention on both earlier tasks.Source: E Additional Continual Learning… · p. 16, §E.1 · p. 16

E.1 Longer Continual Learning Sequence with ScienceWorld

• In the three-stage ALFWorld→WebShop→ScienceWorld sequence, TSRL GRPO improves retention by 2.9 points on ALFWorld and 2.6 on WebShop versus dense SFT→RL, while slightly improving final ScienceWorld performance.The DAPO variant likewise improves prior-task metrics and final ScienceWorld score over SFT→DAPO; pure RL remains strongest for retention, with TSRL close behind.Source: §E.1 · p. 16

E.2 Additional Hybrid Baselines and Joint Training

• Iterative-SFT is weaker than TSRL on the new task, LUFFY incurs more forgetting, and fixed-weight joint losses improve acquisition while sacrificing prior-task retention.The comparisons cover both continual-learning directions, as described in Table A3.Source: §E.2 · pp. 16–17, Table A3 · p. 17

E.3 Performance on Qwen3-4B Model

• On Qwen3-4B, TSRL improves retention over SFT→GRPO by 6.7 points on ALFWorld and 5.6 on WebShop while keeping target-task performance essentially unchanged.Table A4 reports success rates for both continual-learning directions using the GRPO backbone.Source: §E.3 · p. 17, Table A4 · p. 18

F.1 On the Assumption of Querying the Expert

• Expert supervision can come from offline successful-trajectory buffers, cached successes reused as self-distillation, or an online teacher or heuristic solver; the assumption is sparse access to informative trajectories.The experiments retrieve demonstrations when the current policy repeatedly fails on an on-policy state.Source: §F.1 · p. 18

F.2 Correlation Between GSR, Instance Difficulty, and Policy Stochasticity

• GSR treats difficulty as agent-relative: repeated samples can reveal whether exploration succeeds, while internal confidence may be miscalibrated and confidently wrong.The authors present outcome-based GSR as a more reliable trigger than confidence- or entropy-based signals for deciding when to inject expert guidance.Source: §F.2 · p. 18

G.1 Datasets

• WebShop has over 1.18 million products and 12,087 purchase instructions; agents search, browse, and filter, receiving continuous rewards based on product-attribute matches.The reward ranges from 0 to 1.Source: §G.1 · p. 18
• ALFWorld evaluates six household-task types through text interactions, with seen and unseen splits and sparse binary task-completion rewards.Its environments parallel the embodied ALFRED dataset.Source: §G.1 · p. 18
• Deep Search combines BrowseComp’s 1,266 challenging questions with 289 Chinese multi-hop questions across 11 domains; both use sparse exact-match rewards.BrowseComp questions require persistent browsing for hard-to-find information across websites; BrowseComp-ZH addresses challenges including fragmented platforms and implicit linguistic patterns.Source: §G.1 · pp. 18–19
• ScienceWorld is used as the third stage of a continual-learning sequence to test forgetting across task transitions, while AIME 24 and AIME 25 measure retention of reasoning abilities.ScienceWorld tasks draw on elementary-school science; each AIME exam has 15 multi-step problems with integer answers.Source: §G.1 · p. 19

G.2 Training Details

• WebShop and ALFWorld experiments use Verl-Agent, and the additional 7B continual-learning experiments reuse the training recipe unless otherwise stated.That recipe includes rollout group size, trigger threshold, and annealing schedule; Verl-Agent supports multi-turn RL training and modular agent configurations.Source: §G.2 · p. 19

G.3 Prompts

• Deep Search uses Bing Search and a web viewer, with a training configuration adapted to the complexity of multi-hop retrieval tasks.The appendix presents prompts for WebShop, ALFWorld, and Deep Search in Figures A1–A3, and hyperparameter tables for WebShop and ALFWorld and for Deep Search in Tables A5 and A6.Source: §G.2 · p. 19, §G.3 · p. 19, Figure A1 · p. 20