GTR-Turbo: Merged Checkpoint is Secretly a Free Teacher for Agentic VLM Training

1. Introduction

• Multi-turn VLM reinforcement learning can suffer from sparse rewards, long-horizon trajectories, and noisy environments, degrading response and action quality.These problems are associated with incomplete, inconsistent, low-diversity behavior described as thought collapse or entropy collapse.Source: §1 · p. 1
• GTR-Turbo merges historical RL checkpoints into a free teacher, avoiding extra teacher training and dependence on external models.The paper says this preserves GTR’s automated reward mechanism, flexibility, and final performance while accelerating training and reducing compute overhead.Source: §1 · p. 1
• Teacher guidance can be applied through SFT-based online imitation or KL-regularized soft-logit distillation, which uses one forward pass instead of autoregressive teacher generation.The paper presents these as alternatives for subsequent RL training.Source: §1 · p. 2
• GTR-Turbo achieves state-of-the-art results on Points24 and cuts training time by 50%, API calling by 100%, and compute cost by 60%.On image-based ALFWorld, it also makes rapid and stable progress, outperforming many baselines with comparable model size.Source: §1 · p. 2

2.1. Agent Training for LLMs and VLMs

• Agent training has moved from prompting and adapters toward RL for dynamic tasks, with RL4VLM [70] a visual-reasoning reference point and multi-turn RL systems emerging [10] [21] [49] [66] [71].Limited model adjustments can struggle in dynamic, complex environments, restricting robustness and adaptability.Source: §2.1 · p. 2

2.2. Process Guidance Providing Dense Rewards

• Process-guidance approaches address sparse-reward credit assignment through reward models, reward decomposition, large-model judging or labeling, and world models.PRMs can require costly human annotations for high-quality data.Source: §2.2 · p. 2

2.3. Model Merging Techniques

• Model merging combines weights across tasks, hyperparameters, or historical checkpoints; methods include Fisher Merging [26], Task Arithmetic [19], TIES-merging [57], and DARE [64].TIES uses trimming and sign elections to mitigate parameter interference.Source: §2.3 · p. 2

3.1. Revisiting Guided Thought Reinforcement

• GTR uses an external VLM to correct reasoning at each RL step and combines reasoning-token SFT with action-token RL, but API teachers impose access, time, and cost constraints.GTR also incorporates DAgger to mitigate distribution shift during dynamic RL training.Source: §3.1 · pp. 2–3, Figure 1 · p. 3
• GTR-Turbo merges historical RL checkpoints into a free teacher, eliminating dependence on external models while retaining GTR’s automated reward mechanism and flexibility.The authors report comparable or superior performance with reduced training time and token cost.Source: §3.1 · p. 3

3.2. Merged Checkpoints as the Teacher

• TIES merges buffered checkpoints by trimming low-magnitude changes, electing parameter signs, and selectively averaging only sign-aligned parameters.This procedure is intended to mitigate harmful parameter interference from redundant changes.Source: §3.2 · pp. 3–4
• On Points24, merged checkpoints are more stable and perform better than current checkpoints, supporting their use as teachers.The comparison uses a Qwen2.5-VL-7B checkpoint trajectory.Source: §3.2 · p. 3, Figure 2 · p. 4

3.3. Thought Guidance via Supervised Fine-tuning

• GTR-Turbo (SFT) gives the agent and homologous merged teacher the same context, then uses reference thoughts from a replay buffer for SFT alongside PPO.The method also incorporates format rewards and DAgger to stabilize training.Source: §3.3 · p. 4

3.4. Soft Logit Distillation via Minimizing Reverse KL Divergence

• Soft logit distillation replaces autoregressive teacher generation with parallel inference, using negative reverse KL as an auxiliary PPO reward.Because sentence-level KL estimates can be negative, the method clips negative reward parts.Source: §3.4 · p. 4
• KL guidance uses probability information over candidate tokens rather than one-hot labels, requires one forward pass, and removes GTR’s additional thought-dataset memory requirement.The passage characterizes the logit-grounded guidance as almost unhackable.Source: §3.4 · p. 4

4.1. Experimental Setup

• Experiments use Points24 and ALFWorld, sparse-reward visual-agent benchmarks with tasks exceeding 10 and 50 steps, respectively.Training runs use Qwen2.5-VL-7B for 30,000 Points24 steps and 20,000 ALFWorld steps.Source: §4.1 · pp. 4–5

4.2. Effectiveness of the GTR-Turbo Framework

• GTR-Turbo leads on Points24 and matches GTR’s task success rate on ALFWorld, despite lacking access to GTR’s extensive external knowledge.The Points24 evaluation reports the best performance; the ALFWorld comparison reports the same task success rate as GTR.Source: Table 2 · p. 5, §4.2 · p. 6, Table 3 · p. 6
• On Points24, RL4VLM collapses, while GTR improves early; GTR-Turbo improves steadily and its KL-guided variant surpasses all baselines.The SFT-guided variant reaches performance comparable to GTR.Source: §4.2 · pp. 5–6
• On ALFWorld, GTR-Turbo learns from exploration under merged-teacher guidance, achieving scores on par with GTR with better efficiency.The comparison is described as unfair because GTR has access to advanced API models that provide richer guidance.Source: §4.2 · p. 6, Table 3 · p. 6
• The KL-guided variant reduces training time to roughly half of GTR’s and computational cost to as low as 40% of GTR’s.The reported cost covers additional training-mechanism overhead, excluding base agent fine-tuning costs.Source: Table 4 · p. 6, §4.2 · p. 7

4.3. Ablation Studies and Discussions

• A static initial-model KL reference does not improve stably like GTR-Turbo, while rejection sampling cannot solve Points24; RL can explore actions using reward feedback.Rejection sampling trains on self-generated successful trajectories without external models.Source: §4.3 · p. 7
• TIES merging mitigates redundant-parameter interference and sign disagreement; EMA performs best with α = 0.5, comparable to SMA.Very low α degrades early performance when few checkpoints make the recursive merge biased; excessively high α diminishes historical influence.Source: §4.3 · pp. 7–8
• Clipping yields the best KL-estimation results among tested methods, while all non-negative estimators improve performance and forward KL underperforms reverse KL.Clipping controls KL magnitude for finer-grained, more stable updates; reverse KL’s mode-seeking behavior targets a specific teacher peak.Source: §4.3 · pp. 7–8

5. Conclusions and Limitations

• GTR-Turbo uses merged checkpoints as a free teacher for SFT or KL guidance, reducing training time and cost while achieving comparable or superior performance to GTR.The approach is feasible when external teachers are unavailable or data cannot be uploaded for privacy reasons.Source: §5 · p. 8
• When the base model’s initial success rate is below 5%, self-improvement can fail; experiments primarily use 7B models, leaving other model sizes for future verification.In such extreme cases, the authors say traditional GTR is needed because a stronger external teacher can provide better guidance.Source: §5 · p. 8

A. Pseudocodes

• The SFT pseudocode initializes checkpoint and thought buffers, generates teacher reference thoughts, collects environment interactions, then updates the agent with PPO and SFT losses.The updated checkpoint is added to the buffer for subsequent training.Source: A. Pseudocodes · p. 12
• The KL variant’s pseudocode initializes a checkpoint buffer and collects on-policy environment data before computing PPO updates.The supplied pseudocode fragments do not specify further KL-variant steps.Source: A. Pseudocodes · p. 12

B.1. Training Setting

• One epoch of supervised fine-tuning on Qwen2.5-VL precedes RL to give the agent basic instruction-following ability.Points24 labels came from a task solver, while ALFWorld labels were generated by GPT-4V.Source: §B.1 · p. 13

B.2. Hyperparameters

• The authors use LoRA to fine-tune the entire VLM model.The hyperparameters are primarily derived from prior work [51] [70].Source: §B.2 · p. 13
• Table 5 presents the hyperparameters used for GTR-Turbo training.Source: Table 5 · p. 13

• GTR-Turbo remains compatible with Qwen3-VL-8B-Instruct on ALFWorld, and its success rate exceeds that of Qwen2.5-VL-32B.The authors attribute the improvement to Qwen3-VL’s stronger base capability.Source: §C.1 · p. 14, Figure 11 · p. 14
• On ALFWorld, Qwen3-VL can perform RL without SFT initialization, which the authors suggest may simplify GTR-Turbo as foundation models evolve.Source: §C.1 · p. 14

C.2. Additional Experiments on other open-ended tasks

• On Android-in-the-Wild, GTR-Turbo outperforms DigiRL and PPO without heavy hyperparameter tuning or reward shaping.The authors also compare PPO and GTR-Turbo reasoning traces using GPT-5.2 as an LLM judge.Source: §C.2 · p. 14, Table 6 · p. 14

C.3. Pass@k Comparison

• The GTR-Turbo-trained agent surpasses the base model’s pass@k success-rate ceiling, with k=32 converging to the same performance as k=16.Source: §C.3 · p. 14

C.4. Ablation study regarding merging frequency

• GTR-Turbo continues to yield appealing results with merging intervals up to 10, demonstrating robustness to merging frequency.The passage reports this as the merging-frequency ablation shown in Figure 13.Source: §C.4 · p. 14

C.5. Reasoning Score Evaluation

• Without merged-teacher guidance, the RL-trained agent exhibits thought collapse; the merged checkpoint is judged a better reasoning teacher than a regularized reference.Reasoning traces were evaluated for factual correctness, logical rigor, and coherence using GPT-5.2 as judge.Source: §C.5 · pp. 14–15

D. Additional Details on Environments

• Appendix D provides a detailed introduction to the experimental environments.Source: D. Additional Details on Environments · p. 15

D.1. Points24

• Points24 requires constructing a formula equal to 24 from four cards; illegal actions are penalized, and some card hands have no feasible solution.The episode ends when the agent outputs “=” or exceeds 20 steps; a valid formula receives 10, otherwise termination yields -1.Source: §D.1 · p. 15

D.2. ALFWorld

• ALFWorld inputs combine RGB images with action history; removing scene text, as in GTR, increases visual-recognition and long-horizon decision demands.The action space comprises scenario interactions such as moving, taking, placing, opening, and toggling.Source: §D.2 · p. 15
• ALFWorld rewards final-goal completion with 50, subgoal completion with 1, and penalizes inadmissible actions.The reward function specifies these components for each transition.Source: §D.2 · p. 15