PaperlayerView exact revision on arXiv

Source-linked AI summary

ActionOPD: Distilling Decisions, Not Tokens

Huining Yuan, Xiangmin Yi, Zelai Xu, Zixu Wang, Xinlei Chen, Xiao-Ping Zhang, Yu Wang, Chao Yu

arXiv:2610.19820v1cs.AI

TL;DR

Agent distillation must align teacher and student behavior despite choices about which states to supervise and at what granularity, while existing token-based approaches have different limitations. ActionOPD uses semantic agreement between sampled teacher actions and student responses at student-encountered states. Across four out-of-distribution benchmarks, tested configurations improve average performance by 3.3–6.9 percentage points over their corresponding base models.

  • Problem

    Agent distillation must determine which states to supervise and at what granularity, while offline distillation can face state-distribution mismatch and token-level OPD typically requires teacher token probabilities and compatible tokenization.

    Source: §1 · p. 1

  • Method

    ActionOPD samples teacher actions and student responses at student-encountered states, rewarding responses for semantic agreement without requiring teacher token probabilities or tokenizer alignment.

    Source: §1 · p. 2

  • Results

    Across four out-of-distribution benchmarks, all tested configurations improve the four-benchmark average over their corresponding base models by 3.3–6.9 percentage points.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings support semantic actions as a unit of supervision for agent distillation.

    Source: §5 · p. 9

    Teacher errors can become learning targets, semantic matching can misassign rewards, and larger sample sets’ benefit when plausible actions are dispersed remains untested despite added cost.

    Source: §5 · p. 9

Abstract

from arXiv · show

Distilling capable language-model agents into smaller models requires aligning the student's behavior with the teacher's. This raises a fundamental question: at which states and at what granularity should this alignment be performed? Offline imitation provides dense but token-specific supervision and can suffer from state-distribution mismatch. Token-level on-policy distillation (OPD) addresses this mismatch but typically requires teacher token probabilities and compatible tokenization. We argue that semantic actions provide a natural unit of supervision for agent distillation. We introduce ActionOPD, an action-level on-policy distillation method motivated by maximizing expected teacher-action likelihood at student-encountered states. At each such state, ActionOPD samples actions from both the student and a black-box teacher, rewarding student actions according to their semantic agreement with the sampled teacher actions. Through policy optimization, ActionOPD aligns the student's action-level behavior with the teacher's decisions without requiring reproduction of teacher-written reasoning or access to teacher token probabilities. Training Qwen3-4B and Qwen3-8B with two teachers on diverse, curated tool-use tasks yields average gains of 3.3-6.9 percentage points across four out-of-distribution benchmarks. ActionOPD scales favorably with student size and teacher capability, without requiring the teacher and student to share a model family or tokenizer. On $\tau^2$-bench, ActionOPD achieves competitive performance against offline distillation and token-level OPD. We further explore Self-ActionOPD with a same-model teacher and privileged context. These results establish ActionOPD as a promising approach to agent distillation.

1 INTRODUCTION

Agent distillation must choose both which states to supervise and how finely to align behavior, since agent decisions shape later states. ActionOPD aligns semantic actions at student-encountered states; tested configurations improve four-benchmark averages, and the method is competitive with offline distillation and token-level OPD.

1 INTRODUCTION

  • Offline distillation fits teacher-generated sequences but can face state-distribution mismatch, while token-level OPD uses teacher token probabilities and typically requires compatible tokenization.Both approaches supervise tokens without explicitly matching action semantics.Source: §1 · p. 1
  • Semantic actions provide a supervision unit that allows the same decision to be expressed with different wording and reasoning.A tool call conveys a decision through its tool name and arguments, while a natural-language response conveys communicative intent and information.Source: §1 · p. 2
  • ActionOPD rewards semantic agreement between student responses and sampled teacher actions at student-encountered states.The method decouples teacher decision supervision from student token realization.Source: §1 · p. 2
  • Across all tested configurations, ActionOPD improves the four-benchmark average over corresponding base models by 3.3–6.9 percentage points, demonstrating applicability across teacher families and empirical scaling with student size and teacher capability.Source: §1 · p. 2
  • ActionOPD achieves competitive performance against offline distillation and token-level OPD while requiring only generation access to the teacher.It does not require teacher token probabilities or tokenizer alignment.Source: §1 · p. 2

2 ACTIONOPD

ActionOPD aligns student and teacher behavior at student-encountered states using semantic actions rather than matching token sequences. It estimates action agreement from teacher samples and optimizes student responses through state-group rollouts and policy updates.

2.1 FROM ACTION LIKELIHOOD TO POLICY GRADIENTS

  • Action likelihood sums the probabilities of student responses that realize a teacher action, with semantic agreement—not textual identity—as the reward in a REINFORCE-form gradient.The gradient can use sampled teacher actions and student responses without teacher token probabilities or tokenizer alignment; the same reward weights the student's reasoning and action tokens.Source: §2.1 · p. 3

2.2 SEMANTIC MATCHING AND REWARD DESIGN

  • Three sampled teacher responses and a semantic matcher estimate agreement; the matcher checks tool names and canonicalized arguments, then uses intent and factual content for communicative responses.Repeated teacher samples count separately, and c/K estimates the probability that the matcher judges a student response to agree with a sampled teacher response.Source: §2.2 · pp. 3–4
  • The reward gives a unit for the first match and a smaller bonus for additional matches, while invalid outputs receive a separate penalty.With λ = 0.5 and teacher samples [A, A, B], matching A earns 1.25 and matching B earns 1.Source: §2.2 · p. 4

2.3 STATE-GROUP ROLLOUT AND OPTIMIZATION

  • At each state, four student candidates share a teacher multiset; the highest-reward action is committed, while all responses in retained groups contribute to training.Repeated selection creates teacher-guided visited-state trajectories and can explore paths less likely under ordinary student rollouts, with some off-policy state visitation.Source: §2.3 · p. 4, Figure 3 · p. 4
  • Policy optimization uses group-relative advantages and a PPO-style clipped surrogate, excluding groups without supervision or with identical rewards.The group mean is a local baseline, and centered rewards are normalized by a shared batch-wide standard deviation to preserve relative magnitudes across state groups.Source: §2.3 · pp. 4–5
  • These normalized, masked, and clipped updates are not unbiased estimates of the REINFORCE-form gradient.The frequency bonus can still produce different advantages when all candidates match a teacher sample.Source: §2.3 · p. 5

3 EXPERIMENTS

The experiments test ActionOPD’s transfer, scaling, and comparison with other distillation methods, then examine ablations, behavior changes, and self-distillation. Across four out-of-distribution benchmarks, all tested configurations improve over their corresponding base models.

3.1 CROSS-BENCHMARK TRANSFER SETUP

  • The transfer setup trains on 6,697 curated AWM and EnvScaler tasks, then evaluates on four benchmarks excluded from training.The two student sizes are paired with two teachers in a 2×2 design; evaluation uses greedy decoding without teacher-guided action selection.Source: §3.1 · p. 5

3.2 MAIN RESULTS

  • 3.3–6.9 percentage points: all four ActionOPD configurations improve the equally weighted four-benchmark average over their base models on benchmarks excluded from training.The training tasks came from AWM and EnvScaler; evaluation covered BFCL, ToolSandbox, AppWorld, and τ 2-bench.Source: §3.2 · p. 5, §3.1 · p. 5, Table 1 · p. 6
  • With either teacher, Qwen3-8B achieves a higher final average than Qwen3-4B under the same task-start and training-step budgets.The two tested sizes used the same candidate count, teacher-sample count, and reward formulation.Source: §3.2 · p. 6
  • DeepSeek-V4-Flash yields higher student averages than Qwen3.7-Flash at both sizes, with effective transfer across model families and without tokenizer alignment.DeepSeek-V4-Flash also has a higher direct-evaluation average than Qwen3.7-Flash.Source: §3.2 · p. 6

3.3 COMPARISON WITH DISTILLATION BASELINES

  • Against offline distillation and token-level OPD on Tau, ActionOPD exceeds OPD’s test average pass1 by 0.4 percentage points, while KD leads test average pass4 at 19.2.ActionOPD’s test average pass4 is lower than OPD’s, so the results do not establish uniform superiority; its generation-only teacher interface needs neither token log-probabilities nor tokenizer alignment.Source: §3.3 · p. 6, Table 2 · p. 7

3.4 ABLATION STUDIES

  • ActionOPD outperforms Validity Reward, Binary Reward, and Random Commit in average single-trial success on both training and held-out splits.Validity Reward has the largest drop: identical rewards for all valid candidates provide no learning signal in all-valid groups, whereas semantic rewards distinguish candidates by teacher-action agreement.Source: §3.4 · p. 7, Table 3 · p. 7

3.5 PATTERN ANALYSIS

  • Selected matched cases show the student respecting policy constraints, distinguishing information sources from operation targets, and following through on promised actions.The cases pair Base failures with ActionOPD successes on the same task and trial.Source: §3.5 · p. 7, Table 4 · p. 8

3.6 SELF-ACTIONOPD

  • After 50 steps, Self-ActionOPD improves test average pass1/pass4 over Base by 4.8/2.5 percentage points, with single-trial success improving across all three domains.The trained student is evaluated using only public context, unlike the privileged-teacher reference.Source: §3.6 · p. 8, Table 5 · p. 8

4 RELATED WORK

Related work spans token-level distillation, black-box and semantic supervision, and interactive agent learning. ActionOPD’s stated distinction is direct agreement scoring between sampled same-state actions, rather than token distributions, trajectory returns, or correctness judgments.

Distillation and self-distillation

  • Offline imitation fits fixed teacher sequences, while GKD and MiniLLM supervise student-generated sequences through token-level divergences; these methods require teacher distributions or sampled-token log-probabilities.Cross-tokenizer OPD transfers probabilistic supervision through explicit alignment, while OPSD uses a shared-weight teacher with reference-solution context.Source: §4 · p. 9

Black-box and semantic distillation

  • Black-box methods use discriminators, rubrics, verbal scores, or teacher continuations, while ActionOPD matches immediate tool calls or communicative responses against teacher samples at the same state.Its matcher assesses agreement rather than correctness, and the method requires no reasoning-text matching, complete reference trajectories, or terminal rewards.Source: §4 · p. 9

Interactive agent learning and process supervision

  • Interactive methods include expert queries at learner-visited states, local teacher corrections, rollout curricula, and return-derived advantages; ActionOPD scores same-state candidates against a shared teacher multiset without estimating returns.Source: §4 · p. 9

5 CONCLUSION AND LIMITATIONS

ActionOPD uses semantic agreement at student-encountered states to distill teacher decisions, and the reported findings support semantic actions as a unit of supervision. The authors identify risks from teacher errors, imperfect matching, and sparse agreement among fixed-size samples.

5 CONCLUSION AND LIMITATIONS

  • ActionOPD improves average performance on unseen benchmarks, with higher final averages from larger students and stronger teachers, and is competitive with offline distillation and token-level OPD.Ablations support semantic rewards over validity rewards; Self-ActionOPD improves held-out performance with a same-model teacher and privileged context.Source: §5 · p. 9
  • Semantic agreement at student-encountered states decouples teacher decision supervision from student token realization and requires only generation access to the teacher.The authors conclude that these findings support semantic actions as a unit of supervision for agent distillation.Source: §5 · p. 9
  • Teacher errors or suboptimal decisions can become learning targets because agreement does not guarantee correctness.The authors also identify possible false positive or negative matches and sparse matching actions among fixed-size samples, potentially in long-horizon interactions; increasing K and N adds cost, and its benefit in that regime is untested.Source: §5 · p. 9

AI USE STATEMENT

The authors report developing the method and designing all experiments, while using generative AI tools for several research and manuscript tasks. They document language-model use in the experimental protocol and retain responsibility for final research decisions and findings.

AI USE STATEMENT

  • The authors developed the method and designed all experiments.They state that all final research decisions were made by the authors, who take responsibility for the manuscript and findings.Source: AI USE STATEMENT · p. 10
  • Generative AI tools assisted with manuscript preparation, literature review, implementation, debugging, derivation checks, result analysis, and methodological feedback.Source: AI USE STATEMENT · p. 10
  • The experimental protocol documents language-model use for task screening, teacher supervision, semantic matching, and user simulation.Source: AI USE STATEMENT · p. 10

A DERIVATIONS AND IMPLEMENTATION DETAILS

The appendix formalizes ActionOPD's semantic-action objective and describes its supervision, matching, and optimization procedures. Its practical reward and update implementation includes sampling, validity handling, reward normalization, and clipped optimization.

A.1 FROM TEACHER-ACTION LIKELIHOOD TO REINFORCE

  • Student response sequences are partitioned into semantic action classes, with protocol-invalid outputs mapped to an invalid outcome; the induced policy includes that outcome in its action space.Source: §A.1 · p. 14
  • The score-function gradient uses semantic agreement rewards without explicitly evaluating the marginal action likelihood, under fixed-support and bounded-response assumptions.Teacher action samples suffice; teacher token probabilities are not needed. The derivation also notes that subtracting a group mean that includes each candidate's reward scales the otherwise unmodified expected gradient by (N −1)/N.Source: §A.1 · p. 14
  • At each sampled state, the algorithm collects teacher responses and N student responses, rewards the student candidates, executes the highest-reward candidate, then centers and whitens rewards before updating the policy.Source: §A.1 · p. 15

A.2 OBJECTIVE INDUCED BY THE FREQUENCY BONUS

  • The frequency-bonus objective combines expected match count and any-match probability, less invalid-action probability; for 0 < λ < K −1, the first match is rewarded more than each additional match.The objective is conditional on retained samples and does not calibrate teacher probabilities or confidence.Source: §A.2 · p. 15
  • At λ = 0, valid actions receive binary agreement rewards; at λ = K −1, they receive the raw match count, while an empty teacher multiset supplies no supervision.Source: §A.2 · p. 15

A.3 TEACHER SUPERVISION AND SEMANTIC MATCHING

  • Standard ActionOPD gives teacher and student the same retained public context and tool schemas; teacher samples are validity-filtered, and the matcher approximates semantic agreement using tool-call fields or message text.Partial valid teacher samples can supervise a group, but no valid samples yield no student group or policy-failure label; matching errors can make the implemented objective differ from the intended one.Source: §A.3 · pp. 15–16

A.4 OPTIMIZATION AND EXECUTION DETAILS

  • Training retains supervised groups with reward variation, centers and scales rewards across retained groups, and applies advantages to active student-generated tokens before clipped dual optimization.No reference-policy KL penalty or entropy bonus is added.Source: §A.4 · pp. 16–17
  • Collection uses one behavior-policy snapshot, fixes teacher samples before student generation within each group, and overlaps teacher sampling and tool/API waiting across ready states.Source: §A.4 · p. 17

B.1 TRAINING DATA AND TASK SELECTION

The study selects tool-use tasks through context and feasibility checks, then uses transfer-training pools and official splits for specified evaluations. The appendix distinguishes screening for environment feasibility from evidence that a particular policy can succeed.

B.1 TRAINING DATA AND TASK SELECTION

  • AWM task selection excluded initial native prompts exceeding 16,000 tokens under the Qwen3-4B tokenizer, and both data sources underwent static feasibility screening after deterministic integrity checks.Screening assessed whether the environment supported the goal and success conditions, not whether a particular policy could succeed; admission did not require expert rollout success, and static screening does not rule out runtime defects.Source: §B.1 · p. 17
  • Each 100-step transfer run contains 5,900 AWM and 500 EnvScaler task starts, potentially revisiting tasks from a 6,697-task pool; tau comparisons, ablations, and Self-ActionOPD use official 178-task training and 100-task test splits.Source: §B.1 · p. 17
  • Table 6 describes unique-task counts through the task-selection funnel and final environment coverage; Table 7 describes training configurations, with sample counts referring to responses at one state.Source: Table 6 · p. 17, Table 7 · p. 18

B.2 TRAINING CONFIGURATIONS AND AUXILIARY MODELS

Training and generation settings are reported in Tables 7 and 8, with auxiliary models varying by experiment. Tau training and evaluation use different auxiliary models, while Self-ActionOPD also relies on offline-prepared private user summaries.

B.2 TRAINING CONFIGURATIONS AND AUXILIARY MODELS

  • One collection step can include multiple optimizer updates because a PPO epoch processes collected candidates in minibatches.Tau’s 20-decision training limit differs from the environment’s native 200-internal-transition budget.Source: §B.2 · p. 18
  • Tau training uses Qwen3.8-27B for matching and user simulation, while Tau evaluation uses Qwen3.5-9B.Main-experiment matcher and EnvScaler user-simulator settings are shared across all four student–teacher configurations.Source: §B.2 · p. 18
  • Self-ActionOPD’s private user summaries were prepared offline with DeepSeek-V4-Flash and reviewed, leaving auxiliary models in its training setup.Source: §B.2 · p. 18
  • Table 8 reports model roles and generation settings; T, p, and k denote temperature, top-p, and top-k.Response limits are reported in tokens, with 1k denoting 1,024 tokens.Source: §B.2 · p. 18

B.3 EVALUATION PROTOCOL AND SCORE DEFINITIONS

The evaluation uses benchmark-specific task sets, interaction limits, and scoring procedures across BFCL, ToolSandbox, AppWorld, and Tau. The reported average weights benchmark families equally, while repeated training runs are not used to estimate training variation.

B.3 EVALUATION PROTOCOL AND SCORE DEFINITIONS

  • ToolSandbox similarity averages all 1,960 scheduled trials, while BFCL uses 800 multi-turn tasks and one trial per task.ToolSandbox uses 490 strictly local scenarios with four scheduled trials each; BFCL has 200 tasks per category and a native per-turn limit of 20.Source: §B.3 · p. 18
  • AppWorld evaluation covers 168 Normal and 417 Challenge tasks, with one trial, 50 steps, and a 100,000-output-token budget per task.Normal and Challenge scores are separate; the average uses TGC only, where TGC is the fraction passing all task-goal tests.Source: §B.3 · pp. 18–19
  • Tau transfer evaluation covers all 50 Airline, 114 Retail, and 114 Telecom tasks, with four trials per task; students were trained on AWM/EnvScaler, not these tasks.All Tau evaluations disable natural-language assertions, and evaluator success labels do not certify every utterance.Source: §B.3 · p. 19
  • Tau training-set evaluation measures in-domain fit rather than held-out generalization, and the Privileged Teacher receives additional information.Its episodes differ from training rollouts, and ordinary student evaluation excludes reference actions and private task criteria.Source: §B.3 · p. 19
  • Tau pass4 is the fraction of tasks successful in all four scheduled trials, and the main average weights the four benchmarks equally.Tau scores average domain pass1; repeated evaluation trials measure interaction variability, not variation across independent training runs.Source: §B.3 · p. 20

B.4 TAU BASELINES, ABLATIONS, AND COLLECTION BUDGETS

The Tau comparison specifies ActionOPD, reward and commitment variants, and offline or token-level baselines under a shared training setup. Self-ActionOPD uses teacher-side guidance and privileged task information that are withheld from the student and matcher.

B.4 TAU BASELINES, ABLATIONS, AND COLLECTION BUDGETS

  • ActionOPD’s reference Tau configuration uses Equation 6 with K = 3, λ = 0.5, and uniform tie-breaking among maximum-reward candidates.The compared configurations share the 178-task training set, Qwen3-4B initialization, and step-50 reporting point.Source: §B.4 · p. 20
  • Validity Reward replaces semantic training rewards with +1 for valid actions and −1 for invalid outputs or transfer-protocol violations.Teacher sampling and semantic matching remain active for ActionOPD’s commitment rule, and all-valid groups are masked.Source: §B.4 · p. 20
  • Binary Reward gives +1 for any match, zero for valid non-matches, and −1 for invalid actions; Random Commit selects uniformly among all four candidates.Binary Reward uses λ = 0; Random Commit has no success-based resampling and retains all candidates in retained groups for training.Source: §B.4 · p. 20
  • KD trains on successful teacher-generated trajectories, including their reasoning and action/message tokens, while OPD applies sampled-token reverse-KL supervision to student trajectories.OPD requires teacher log-probabilities and a compatible token space; KD filters demonstrations by outcome labels rather than using policy-gradient rewards.Source: §B.4 · p. 20
  • Equal task starts and collection steps do not equalize retained examples, tokens, teacher queries, or compute across KD, OPD, and ActionOPD.KD and OPD collect four trajectories per task start; ActionOPD samples four candidates per state and executes one, while retained-group candidates contribute to training.Source: §B.4 · p. 21

B.5 SELF-ACTIONOPD: TEACHER INPUTS AND PARAMETER UPDATES

  • Self-ActionOPD uses Qwen3-4B in both roles, with student candidates and teacher samples drawn from the same current parameters before optimization.Teacher samples and matching remain fixed during optimization; subsequent collection uses updated shared parameters.Source: §B.5 · p. 21
  • The Self-ActionOPD teacher receives privileged summaries and reference task information withheld from the student and matcher, although this information can affect which student actions are rewarded.The trained student is evaluated without privileged inputs; teacher guidance also covers immediate actions, preconditions, authorization, clarification, and avoiding repetition.Source: §B.5 · p. 21

Related papers

Loading 2610.19820v1…