Function-Space Temporal Self-Distillation for Policy Optimization

1 INTRODUCTION

• Scalar outcome rewards make credit assignment difficult because they compress decisions of varying quality across a response into one outcome.Self-distillation instead compares unassisted and feedback-informed views to provide token-level supervision.Source: §1 · p. 1
• A continuously evolving self-teacher creates moving targets across training steps, motivating stabilization that can add persistent parameter-copy memory overhead.SDPO also does not explicitly distinguish severe locally overvalued actions or adapt supervision to problem-specific reliability.Source: §1 · pp. 1–2
• FSPO stores sparse historical next-token distributions as fixed temporal targets, separating supervision across training steps without a persistent EMA teacher.It combines those targets with current privileged adaptation and replays historical behaviors for consolidation.Source: §1 · p. 2
• FSPO suppresses actions strongly preferred by the unassisted policy but rejected by the privileged view, while adapting reference supervision to policy reliability.It strengthens reference loss after repeated failures and reduces supervision as reliable solutions emerge; quality- and coverage-aware memory retains diverse, nonredundant behaviors.Source: §1 · p. 2
• Approximately 4% points higher average accuracy than the strongest baseline, with peak memory reduced from about 70 GB to 56 GB on Olmo3-7B-Instruct.Figure 1 reports the highest accuracy with the fewest generated tokens across five tasks.Source: Figure 1 · p. 1, §1 · p. 2

2 FUNCTION-SPACE POLICY OPTIMIZATION

• FSPO preserves selected historical privileged predictions as sparse, fixed next-token targets, while combining them with current adaptation in a single-policy framework.Unlike current privileged targets, the historical targets remain fixed as the policy evolves.Source: §2 · p. 3

2.1 PRIVILEGED ADAPTATION AND DISAGREEMENT CORRECTION

• Privileged adaptation supplies dense token-level guidance and explicitly corrects tokens the student overvalues relative to privileged evidence.The privileged demonstration prioritizes a successful sibling response, then a retained trajectory, and finally the verified reference; distributions are restricted to top-K student tokens plus pooled tail mass.Source: §2.1 · pp. 3–4

2.2 FUNCTION-SPACE TEMPORAL CONSOLIDATION

• Temporal memory retains trajectories using verifier quality and structural complementarity, then consolidates their fixed historical targets as the policy evolves.Targets become eligible for refresh after H steps, with a fraction refreshed every F steps to limit staleness while preserving targets of different ages.Source: §2.2 · pp. 4–5

2.3 RELIABILITY-ADAPTIVE REFERENCE GROUNDING

• Reference supervision is weighted according to self-generated reliability, strengthening when the policy repeatedly fails and decreasing as reliable behavior emerges.Source: §2.3 · p. 5

2.4 UNIFIED ADAPTATION AND CONSOLIDATION OBJECTIVE

• The unified objective periodically shifts emphasis from current privileged adaptation toward historical consolidation, with memory pre-seeded so consolidation starts immediately.Table 1 compares highest avg@16 within 1-hour and 5-hour budgets for Qwen3-8B and Olmo3-7B-Instruct; its final column averages performance across five benchmarks at 5 hours.Source: §2.4 · p. 6, Table 1 · p. 6

3.1 EXPERIMENTAL SETTING

• The evaluation uses two model backbones, four scientific reasoning subsets, and ToolAlpaca, with held-out avg@16 measured under shared training settings and budgets.Methods use the same initialization and hardware budget; the highest accuracy within 1-hour and 5-hour runs is reported.Source: §3.1 · pp. 6–7, §3 · p. 6

3.2 MAIN RESULTS

• FSPO achieves the highest Avg@5h on both model backbones and every scientific reasoning task, while remaining competitive on Tool Use.On Tool Use, it matches the strongest Qwen3-8B result and improves over baselines on Olmo3-7B-Instruct; SDPO is stronger at 1 hour on both backbones.Source: §3.2 · p. 7

3.3 LEARNING WITHOUT FORGETTING

• FSPO improves trained-task performance while retaining prior capabilities better than baselines across five prior-capability benchmarks.The authors attribute this retention to replaying fixed historical predictions through function-space temporal consolidation.Source: §3.3 · pp. 7–8

3.4 MEMORY AND COMPUTATIONAL PROPERTIES

• FSPO reaches about 56 GB peak memory on Olmo3-7B-Instruct, versus 70 GB for GRPO and SDPO, by avoiding extra persistent model copies.The cache peak is about 5 MB; temporal replay adds activations, and the projected absolute memory gap grows with model size.Source: §3.4 · p. 8, Figure 6 · p. 8

3.5 ABLATION STUDY

• Across Biology and Tool Use, the full FSPO components raise average accuracy from 27.8% for the zero-shot base model to 64.5%.Temporal consolidation and reference grounding account for the largest gains, particularly on Tool Use; disagreement correction adds a smaller but consistent improvement. Sparse function-space targets, quality-and-coverage memory, adaptive grounding, and cyclic adaptation followed by consolidation outperform their tested alternatives.Source: §3.5 · pp. 8–9

4 RELATED WORK

• RL methods address coarse trajectory-level credit through step-level verification, reward redistribution, selective updates, and richer feedback or self-training.On-policy self-distillation conditions the same policy on references, demonstrations, or feedback, but its evolving teacher can suppress useful uncertainty.Source: §4 · p. 9
• Evolving distillation targets are stabilized with target networks, EMA teachers, trust regions, or expert iteration, while continual learning uses regularization, output distillation, and replay.FTSD adapts function-space output preservation to LLM self-distillation using sparse privileged distributions during training rather than persistent model copies.Source: §4 · p. 9
• Knowledge distillation spans sequence-level and on-policy language-model methods, with student-generated trajectories reducing train–inference mismatch.Prior function-space approaches preserve previous outputs, including stored-logit replay.Source: §4 · p. 9

5 CONCLUSION

• FSPO preserves historical behavior with sparse function-space targets rather than a persistent teacher model.It combines privileged adaptation with disagreement correction, reliability-adaptive grounding, and quality- and coverage-aware temporal memory.Source: §5 · p. 10
• Across scientific reasoning and tool-use tasks on two backbones, FSPO achieves the highest average accuracy under a 5-hour budget and better preserves prior capabilities.The conclusion also reports faster convergence and peak memory reduced from about 70 GB to 56 GB on Olmo3-7B-Instruct.Source: §5 · p. 10
• The ablations indicate that the reported gains arise from interaction among current adaptation, temporal consolidation, and adaptive grounding rather than a single component.Source: §5 · p. 10

• Continual learning and experience replay are represented among the cited works.Source: H. Lightman, V. Kosaraju, Yura Burda,… · p. 13
• Several cited works address reinforcement learning, including trust-region policy optimization and proximal policy optimization.Source: H. Lightman, V. Kosaraju, Yura Burda,… · p. 14
• The references include work on language feedback, self-distillation, and verbal feedback without scalar rewards.Source: H. Lightman, V. Kosaraju, Yura Burda,… · pp. 13, 15, 16

A.1 COMPLETE TRAINING PROCEDURE

• FSPO explores, learns from current privileged evidence, grounds on the reference when unreliable, consolidates fixed historical targets, updates the policy, and refreshes memory.The objective combines current and temporal losses with reference cross-entropy, using a cyclic consolidation weight.Source: §A.1 · p. 17
• Memory stores retained trajectories, verifier scores, structural signatures, and sparse fixed targets; verified rollouts replace entries with lower quality-and-coverage retention scores.Reference-seeded entries are fallback targets, not verified demonstrations; stored targets use the current policy and remain fixed in host memory.Source: §A.1 · p. 17
• Current privileged learning selects demonstrations in order: verified sibling, retained trajectory, then reference solution.The policy compares unassisted and privileged-conditioned next-token distributions and adds a penalty for overvalued tokens.Source: §A.1 · p. 17
• Reference grounding uses a smoothed reliability estimate to weight reference cross-entropy when the policy remains unreliable.Source: §A.1 · p. 17

A.2 ADDITIONAL DETAILS ON CATASTROPHIC FORGETTING

• FSPO achieves the highest average retention score for every adaptation dataset, with scientific-task averages near or above the base score of 61.26.Chemistry and Biology score 60.95 and 60.94; Physics and Materials score 62.61 and 62.32.Source: §A.2 · p. 18
• 59.45 versus 61.26: after Tool-use training, FSPO’s average retention trails the base model but exceeds GRPO, on-policy GRPO, and SDPO.The respective comparison scores are 54.79, 57.58, and 58.73.Source: §A.2 · p. 18
• HumanEval improves from 47.56 in the base model to 51.83–58.54 after scientific adaptation, while HellaSwag, Winogrande, and TruthfulQA generally remain below base scores.MMLU remains close to its base score of 58.77 and slightly exceeds it after Materials and Tool-use adaptation.Source: §A.2 · p. 18

Rollout size and matching objective

• G = 8 provides enough rollouts to find verified siblings for most problems, while G = 16 adds no gain under the fixed wall-clock budget.Symmetric JSD outperforms forward and reverse KL; moderate disagreement correction helps, but λneg = 1.0 underperforms removing the correction.Source: §A.3 · pp. 19–20

Target refresh

• H = 4 with ζ = 0.5 performs best: refreshing too early weakens temporal separation, while no refresh leaves targets stale.Source: §A.3 · p. 20

Grounding and schedule parameters

• Smoothing reliability matters; ν = 1 degrades performance, and τ = 1.0 keeps grounding active even on problems the policy already solves.Strong grounding can override useful self-generated behavior; λCE = 1.0 falls close to the distillation-only variant.Source: §A.3 · pp. 20–21

Training dynamics

• Figure 7 tracks five-task mean training dynamics, including objective contributions, guidance assignments, divergence, cache growth, rewards, solve rate, and active-token length.Task metrics are held at their final values after a task’s run ends; reported variation is mean ± 1 σ.Source: Figure 7 · p. 21

A.4 TRAINING CONFIGURATION

• FSPO uses one resident policy for rollouts and privileged evaluation, requiring neither an EMA teacher nor rollout importance-sampling correction.Backbones, data limits, batching, rollout sampling, validation, distillation objective, and optimizer follow the SDPO setting.Source: §A.4 · p. 21

A.5 TRAINING DYNAMICS

• Reference grounding contributes strongly early and falls as the policy produces reliable evidence, while temporal consolidation remains active longer through disagreement with cached targets.The cyclic ρs schedule later rebalances adaptation and consolidation.Source: §A.5 · p. 22
• Verified siblings soon dominate privileged guidance; reference answers decline as a fallback, while retained trajectories are used when no verified sibling is available.Occasional later fallback spikes reflect isolated batches without verified siblings.Source: §A.5 · p. 22
• Temporal JSD decays more slowly than current JSD, preserving historical correction after current predictions align; cache signature entropy plateaus near ln 4, indicating balance across answer options.On these tasks, the signature is the predicted option.Source: §A.5 · p. 22
• The cache grows from seeded reference trajectories and sparse teacher targets, with ready entries closely tracking total entries as verified behaviors are admitted or replaced.Completed runs hold their final cache sizes, flattening the cross-task average as other tasks continue to grow.Source: §A.5 · p. 23
• 942 mean entries occupy 4.8 MB across five tasks, and peak cache use is about 5 MB of CPU memory.Entries store sparse function-space information rather than model-sized state.Source: §A.5 · p. 23
• Mean cache reward rises toward a stable high-quality regime as active-token length contracts after an initial spike and then stabilizes.The early length spike reflects longer trajectories from the unadapted policy; the later pattern accompanies increasingly compact generations.Source: §A.5 · p. 23