Source-linked AI summary
Persistent Teacher Anchoring for Tool-Using Agents
Hyun Bin Park, Kyungho Song, Sangmin Lee, Du-Seong Chang
TL;DR
On-policy distillation leaves tool execution outside proposer-verifier control, so student calls can execute before supervision and shape later prefixes. PTA adds turn-level commitment to chunk verification and uses persistent atomic states for lookahead scheduling. Under matched downstream RL budgets, it improves macro best@4 over OPKD in retrieval and perception while increasing throughput through lookahead.
Problem
Existing proposer-verifier formulations govern retained text but leave tool execution and observation treatment unspecified, despite student calls executing before teacher supervision.
Method
PTA has the student propose chunks, the teacher verify them, and the runtime execute a tool only after the complete assistant turn is committed; fixed-verifier atomic states support persistent lookahead.
Results
Under the same downstream RL budget, PTA improves macro best@4 over OPKD by 2.5 points in retrieval-mediated reasoning and 2.8 points in perception-mediated reasoning.
Takeaways & Limitations
PTA gives downstream RL an initial policy shaped by teacher-verified tool interactions, while lookahead improves scheduler throughput by 24%.
Takeaways & Limitations
PTA requires teacher inference during rollout, and its verifier-cost trade-off may change with model scale, tool latency, and task horizon.
Abstract
from arXiv · showhide
Distillation is common in LLM post-training, where on-policy knowledge distillation (OPKD) uses student-generated trajectories to prepare the student for downstream RL. At each state, the student matches a next-token distribution supplied by the teacher. As the rollout enters states the teacher would not visit, the teacher-student distribution gap can accumulate. In tool use, this gap becomes consequential because student-written calls execute before supervision and their observations shape later prefixes. Proposer-verifier generation addresses this drift by letting the teacher decide which student-proposed text is retained during generation. Existing formulations govern text but leave tool execution outside their scope. We propose Persistent Teacher Anchoring (PTA), a student-induced but teacher-committed rollout construction. PTA retains chunk-level verification and adds turn-level commitment, allowing a call to reach the environment only after the teacher has verified the entire turn. Treating verified chunks as atomic generation units, we introduce persistent lookahead, which fills idle rollout capacity by advancing future samples and carrying unfinished ones across student updates under the fixed verifier. Across Search-R1-style retrieval and DeepEyes-style perception RL, applying PTA before downstream RL improves macro best@4 by 2.5 and 2.8 points over OPKD under the same downstream RL budget, while lookahead improves throughput by 24%.
1 Introduction
PTA extends proposer-verifier distillation to tool execution by requiring teacher-verified, committed turns before calls reach the environment. Its persistent verified states enable lookahead scheduling, and experiments report gains over OPKD under matched downstream RL budgets.
- Motivation: Tool-use distillation is risky because student-generated calls execute before teacher supervision and their observations alter later prefixes.Proposer-verifier methods address distribution drift during generation but traditionally specify no execution rule for tool calls.
- PTA: PTA lets the student propose chunks while the teacher verifies them, then commits the entire turn before any valid tool call executes.Observations remain context, while distillation applies at committed assistant positions.
- Persistent scheduling: Persistent teacher anchoring carries verified atomic states across student updates because the verifier remains fixed.Lookahead uses these states to fill idle rollout slots and carry unfinished work forward.
- Persistent scheduling: 24% throughput improvement comes from lookahead filling idle slots and carrying unfinished rollout work across updates.The construction uses atomic boundaries and a fixed verifier.
- Results: Under the same downstream RL budget, PTA improves macro best@4 over OPKD by 2.5 points in retrieval and 2.8 points in perception.The reported evaluations cover retrieval-mediated reasoning and perception-mediated reasoning.
2 Related Work
Prior work places proposer-verifier decisions in decoding or training and often constructs tool-use trajectories externally. PTA remains on-policy while extending teacher decisions from retained text to tool execution.
- On-Policy Knowledge Distillation: On-policy distillation computes teacher feedback after the student generates a trajectory, typically matching teacher token probabilities over that sequence.No teacher decision enters generation itself.
- Proposer-Verifier Generation: Proposer-verifier methods move the stronger model’s decision into generation, spanning speculative decoding, acceptance judgments, intervention, and chunk verification.These methods differ in stage and proposal unit.
- Tool Execution: Text-only proposer-verifier formulations define what text is retained but not when a generated tool call executes.PTA supplies the missing execution event.
- Tool-Use Distillation: Constructed-trajectory approaches generate or verify trajectories before training and use their tokens as hard targets, whereas PTA remains on-policy.PTA has the student propose every assistant span and matches teacher distributions at committed positions.
- Downstream RL: PTA supplies an initial policy for downstream tool-use RL, including retrieval-mediated reasoning and active visual perception.The cited applications include Search-R1 and DeepEyes-style agents.
3 Method
PTA combines chunk-level teacher verification with turn-level commitment, ensuring that only complete verified turns can execute tools. Verified chunks and completed environment transitions become atomic rollout states for training and persistent lookahead.
- Turn Commitment: PTA extends chunk verification with turn commitment: verified chunks accumulate in a pending turn, which enters the trajectory only at assistant EOS or a tool-call closing tag.Only then does the runtime parse and potentially execute the call.
- Tool-Use Rollouts: Directly applying text-stream verification to tools creates execution-before-verification, correction-after-execution, ungoverned-observation, and undefined-target problems.Observations are produced by the environment and cannot be treated like ordinary proposed text.
- Distillation Objective: PTA keeps prompt, committed assistant, and observation spans in the prefix but applies distillation only at committed assistant positions.Observation tokens are not imitation targets.
- Chunk Verification: The student proposes chunks from the verified prefix, while the teacher accepts top-K-supported tokens or replaces the first rejected token with top-1 and discards its suffix.The verified chunk is the smallest appended unit.
- Atomic States: PTA exposes resumable states only after verified chunks or complete environment transitions, never during partial turns or pending tool calls.A valid committed tool call executes, returns an observation, and then generation resumes.
- Persistent Lookahead: Persistent scheduling reuses these atomic states across student updates because the verifier remains fixed while new chunks still pass through it.The implementation integrates teacher verification into the chunk-generation loop and reuses the same state boundaries for lookahead.
4 Experiments
Experiments compare PTA with OPKD and Direct RL across retrieval- and perception-mediated reasoning under matched downstream RL budgets. PTA improves aggregate outcomes, preserves more teacher probability under sparse distillation, and increases rollout throughput through lookahead.
- Retrieval-mediated reasoning: PTA + RL leads the strongest retrieval aggregates and exceeds OPKD + RL by 2.52 points on macro best@4.It leads weighted best@4, macro mean@4, and macro best@4, while tying Direct RL on weighted mean@4.
- Retrieval-mediated reasoning: The largest retrieval gain occurs on Musique, where mean@4 rises from 8.20 to 10.93 and best@4 from 13.94 to 18.78.The passage connects this concentration with multi-step evidence gathering and later queries shaped by earlier retrieval interactions.
- Perception-mediated reasoning: PTA + RL gains 5.32 points on macro mean@4 and 2.80 points on macro best@4 over OPKD + RL in perception.PTA + RL leads on all four aggregate metrics, although benchmark-level advantages are not uniform across VStar and HRBench4K.
- Alignment before downstream RL: PTA leads OPKD on all six pre-RL perception metrics, with HRBench8K best@4 increasing from 58.00 to 65.00.This comparison is made at the same training step before downstream RL.
- Top-N distribution stability: Teacher-committed prefixes preserve more teacher distribution under the same top-N budget than student-committed OPKD prefixes.PTA maintains high average and minimum retained mass at N = 256, while OPKD shows sharper low-coverage positions even at N = 1024.
- Lookahead efficiency: Lookahead raises throughput from 0.519 to 0.644 samples/s, a 24% increase, by filling otherwise idle rollout capacity.It promotes 20.2 samples per step on average without shortening generation or update time.
5 Conclusion
PTA extends proposer-verifier generation so teacher verification governs both retained text and tool execution. Across retrieval and perception RL, it improves downstream performance and sparse teacher-distribution fidelity, while persistent verified states enable a 24% throughput increase through lookahead.
- Conclusion: PTA makes the teacher decide when a completed assistant turn may execute a tool, with verified rollout states persisting across student updates.The student proposes chunks, the teacher verifies them, and only finalized turns are committed and executed.
- Conclusion: PTA + RL improves macro best@4 over OPKD by 2.5 points in retrieval and 2.8 points in perception under the same downstream RL budget.The experiments also show greater teacher-mass retention under a fixed top-N budget and a 24% scheduler-throughput improvement.
Limitations
The evaluation covers retrieval-mediated and perception-mediated reasoning under matched student and downstream-RL conditions, but excludes several tool-use settings and varies only a specified verifier configuration. PTA also corrects rejected calls before execution and requires teacher inference during rollout construction.
- PTA is evaluated only in retrieval-mediated and perception-mediated reasoning, covering textual and visual tool observations.
- The comparison fixes the student proposer and downstream RL budget while varying whether teacher information enters before or after commitment.
- Code execution, database access, and multi-agent interaction are outside the study’s scope.
- PTA corrects verifier-rejected tool calls before execution, so pre-RL training does not expose the resulting error states.
- The verifier trade-off may change with model scale, tool latency, and task horizon, and teacher inference remains required during rollout construction.
- Experiments use one fixed verifier configuration and one training run per setting, so reported variability reflects evaluation sampling rather than training-seed variation.
A Full Perception-Mediated Results
Detailed perception results show that PTA’s advantage depends on the subtask, metric, resolution, and image relationship being evaluated. PTA leads several mean@4 and higher-resolution cross-image comparisons, while other best@4 and single-image metrics favor competing systems.
- VStar: PTA + RL exceeds OPKD + RL in VStar mean@4 for both attribute and spatial reasoning, while OPKD + RL leads both best@4 subtasks.
- HRBench4K: On HRBench4K, PTA + RL leads benchmark-level mean@4 and both single-image metrics, while Direct RL leads benchmark-level best@4 and both cross-image metrics.
- HRBench8K: On HRBench8K, PTA + RL leads benchmark-level and cross-image metrics, including cross-image mean@4 increasing from 43.75 to 51.50 over OPKD + RL.
- HRBench8K: Direct RL remains higher on HRBench8K’s two single-image metrics, concentrating PTA’s 8K advantage in cross-image perception at higher resolution.
B Significance of the Benchmark-Level Differences
The significance analysis tests benchmark-level mean@4 differences using conservative, unpaired variance estimates and unadjusted two-sided p-values. Three differences reach p < 0.01, while others are not separable from zero at the reported resolution.
- The analysis tests benchmark-level mean@4 differences between PTA + RL and OPKD + RL, where mean@4 averages four sampled rollouts per question.
- Variance is bounded by p̂(1−p̂), using benchmark accuracy and evaluation-question counts under an independence assumption.
- Reported p-values are two-sided and unadjusted for multiple comparisons, and the systems are not paired on the same questions.
- Each setting is trained once, so the significance analysis captures evaluation-sampling variability rather than training-seed variability.
- Three differences reach p < 0.01; Musique and HRBench8K are among the settings associated with the largest reported gains.
- HRBench4K reaches p < 0.1, whereas differences on NQ, HotpotQA, and VStar are not separable from zero at this resolution.
D The Committed-Token Distribution
The committed-token distribution is induced by a deterministic verifier applied to student proposals: teacher-supported tokens are retained, while rejected proposals become the teacher’s top-1 token. Its departure from the student policy equals the rejected mass, with K controlling the interpolation between on-policy and greedy teacher behavior.
- At each verified context, the student proposes a token and the verifier keeps it inside the teacher’s top-K support or replaces it otherwise.
- The committed-token law is the student conditional pushed forward through the deterministic verifier map.
- The resulting distribution mixes truncated student proposals with probability 1−r(h) and the teacher’s top-1 token with probability r(h).
- The committed-token law differs from the student policy only by the teacher-rejected mass, with realized replacement rates of 9.5% in retrieval and 11.6% in perception.
- PTA uses K = 3, while K equal to the full vocabulary yields purely on-policy sampling and K = 1 yields the teacher’s greedy continuation.
- The probability of an entirely unrejected trajectory decreases with trajectory length, giving longer tool-use rollouts more opportunities for verifier intervention.
- The mixture determines supervised positions, while the distillation objective still matches the teacher conditional at committed assistant positions.
E Verifier Acceptance and Correction Statistics
Verifier intervention is widespread across tool calls and chunks, while token-level correction declines during perception training and teacher-side rollout cost falls. These statistics measure intervention frequency and location, not the semantic effect of individual corrections.
- Acceptance and correction rates: 93.0% of perception tool-call tokens are kept, while 84.7% of tool calls contain at least one replacement.Longer perception spans and a call-level correction threshold help explain why many calls are corrected despite high token acceptance.
- Acceptance and correction rates: Among corrected calls, the median number of replaced tokens is five in perception and one in retrieval.
- Training dynamics: Verifier-kept tokens rise from 0.873 to 0.902, while rejected mass falls from 0.127 to 0.098, a 23% reduction.The rejected mass is the total variation distance between the committed-token distribution and the student policy.
- Training dynamics: Chunks receiving at least one replacement decline from 0.975 to 0.941, so correction remains distributed across nearly every chunk.Roughly one token in ten is replaced, rather than corrections concentrating in a few chunks.
- Training cost: Chunks per sample fall from 109 to 77, reducing teacher-verifier calls as accepted runs become longer.Each chunk corresponds to one teacher-verifier call.
- Interpretation boundary: These statistics report intervention frequency and location but do not classify how individual corrections change tool-call semantics.
F Teacher-Gated Carryover under Student Updates
PTA preserves teacher commitment when unfinished rollouts cross student updates. Carryover contains only verified trajectory state or completed environment transitions, and every post-update continuation passes the fixed verifier before affecting the rollout.
- Carryover invariants: Lookahead may carry unfinished rollouts across updates, but PTA does not claim equivalence to regenerating them under the updated student.The limitation is stale-prefix risk, not unverified post-update content entering the rollout.
- Carryover invariants: A carryover state is PTA-consistent only when committed assistant spans, completed observations, and any pending assistant turn contain no raw speculative chunk.Observations enter only after a committed tool call and completed environment transition.
- Post-update continuation: After an update, the carryover state remains unchanged and the updated student may continue generation from it.
- Post-update continuation: A speculative post-update chunk is not appended to the pending turn, committed trajectory, environment, or loss target before verification.
- Post-update continuation: Only the verified chunk is retained, so the fixed teacher verifier gates continuation before it affects rollout state or training targets.This preserves PTA commitment semantics across student updates even when carryover is stale relative to the updated student.
- State representation: The carryover state is represented as Ss = (τs, Rs), with τs the committed trajectory and Rs the pending assistant turn.
G Training Setup
Experiments use asynchronous, multi-turn PPO-based training with GRPO-style advantage estimation and teacher inference on the same four-GPU node. Retrieval and perception settings use distinct model pairs, datasets, tool interactions, reward schemes, and training durations.
- Shared setup: All experiments run on a single node with four GPUs using a PPO-based loop and GRPO-style advantage estimation.Teacher inference uses four replicas on the same node, without a separate validation stage during training.
- Shared setup: Training uses batch size 32, no KL reward, KL loss, or entropy regularization, and zero entropy coefficient.
- Shared setup: Distillation uses truncated forward-KL over the top 256 teacher candidates, with speculative chunks of 128 and top-3 verification during alignment.
- Shared setup: Rollouts are asynchronous, support multi-turn interaction, and use fixed token budgets for student generation and teacher inference.
- Retrieval setting: Search-R1-style training uses Qwen3-1.7B and Qwen3-32B on a preprocessed question-answering dataset with exact-match or optional LLM-as-a-judge rewards.Episodes allow up to 15 assistant and 15 user turns, one tool call at a time, and 2,048-token tool responses.
- Perception setting: DeepEyes training uses Qwen3-VL-2B-Thinking and Qwen3-VL-32B-Thinking on a visual-toolbox dataset with zoom-in interactions.Rewards combine answer accuracy, format consistency, and tool-use behavior; training lasts one epoch.