PaperlayerView exact revision on arXiv

Source-linked AI summary

Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradients

Byung-Kwan Lee, Ximing Lu, Shizhe Diao, Minki Kang, Saurav Muralidharan, Karan Sapra, Andrew Tao, Pavlo Molchanov, Yejin Choi, Yu-Chiang Frank Wang, Ryo Hachiuma

arXiv:2606.18216v1cs.CL

TL;DR

Small students need broad generalization for deployment, but standard distillation becomes limited when the student is much smaller than its teacher. ZPPO keeps teacher guidance in reformulated prompts, outperforming distillation and GRPO across a 31-benchmark suite, with the largest gains at 0.8B.

  • Problem

    Small deployment models need broad generalization, yet logit distillation becomes structurally limited when the student is much smaller than its teacher.

  • Method

    ZPPO keeps teacher guidance in prompts through BCQ and NCQ reformulations for hard questions, preserving the student’s response-level on-policy update.

  • Results

    +9.3/+5.2/+4.0/+2.8 pp on 16 VLM benchmarks and +6.8/+4.3/+2.7/+2.7 pp on 10 LLM and 5 Video benchmarks, ZPPO outperformed distillation.

  • Takeaways & Limitations

    ZPPO’s gains were largest for the smallest students and extended beyond the training corpus to LLM and Video benchmarks.

  • Takeaways & Limitations

    BCQ requires teacher success, so questions missed by both teacher and student receive only the more limited NCQ contribution.

Abstract

from arXiv · show

Knowledge distillation transfers a teacher's competence to a small student but is brittle in the small-student regime: forcing the student to imitate logits from a much larger teacher concentrates it on the teacher's sharpest modes, hurting generalization on benchmark families beyond the training corpus. Reinforcement learning (RL) avoids logit imitation by training on the student's own rollouts. However, on questions where every rollout fails-yielding zero advantage and being silently discarded-injecting a stronger teacher's response into the policy gradient breaks the on-policy assumption and induces drift. We introduce Zone of Proximal Policy Optimization (ZPPO), inspired by Vygotsky's zone of proximal development, which keeps the teacher inside the prompt rather than the policy gradient. On hard questions, ZPPO constructs two reformulated prompts: a Binary Candidate-included Question (BCQ) pairs one correct teacher response with one incorrect student response as anonymized candidates the student must discriminate, and a Negative Candidate-included Question (NCQ) aggregates the student's wrong rollouts into a single prompt to surface their shared failure modes. A prompt replay buffer recirculates each hard question until it either graduates-the student's mean rollout accuracy on it reaches half- or is FIFO-evicted under finite capacity, amplifying BCQ and NCQ inside the student's current zone of proximal development. On the Qwen3.5 family at four student scales (0.8B-9B) with a 27B teacher, post-trained as vision-language models and evaluated on a 31-benchmark suite (16 VLM, 10 LLM, 5 Video), ZPPO outperforms off/on-policy distillation and GRPO, with the largest gains at the smallest scale.

1. Introduction

ZPPO addresses small-model deployment limits by keeping teacher guidance in reformulated prompts rather than logits or policy gradients. Across four student scales and 31 benchmarks, it improves training-aligned and out-of-distribution generalization, with the largest gains at 0.8B.

  • Small-model deployment requires broad generalization under tight compute and memory budgets, but frontier gains largely fail to transfer to mobile, AR/VR, and robotic settings.
  • Logit-based distillation becomes brittle when the student is much smaller, concentrating learning on teacher behavior and hurting generalization beyond the training corpus.
  • RL avoids logit imitation but discards hard questions with zero successful rollouts, while inserting teacher responses breaks on-policy training and induces policy drift.
  • ZPPO keeps the teacher inside prompts: BCQ pairs correct teacher and wrong student candidates, while NCQ aggregates wrong student rollouts to expose shared failures.
  • Hard questions replay until mean rollout accuracy reaches 0.5 or FIFO eviction, focusing practice inside the student’s zone of proximal development.The prompt replay buffer is layered with BCQ and NCQ on top of GRPO-based training.
  • On 16 VLM benchmarks, ZPPO gains +9.3/+5.2/+4.0/+2.8 pp at 0.8B/2B/4B/9B, while on 10 LLM and 5 Video benchmarks it improves +6.8/+4.3/+2.7/+2.7 pp.Distillation averages only +0.9/+0.9/+0.7/+0.6 pp on VLM benchmarks and degrades LLM and Video benchmarks by −2.5/−1.8/−0.9/−0.3 pp.

2. Related Work

Related work spans knowledge distillation, RL post-training, and prompt replay, curriculum, or hint scaffolding. ZPPO differs by moving teacher knowledge into BCQ and NCQ prompts while sampling responses from the student’s own policy.

  • Knowledge distillation: Knowledge distillation uses a shared logit-distribution training signal, which becomes brittle for much smaller students and induces memorization and mode-seeking.This applies across off-policy, on-policy, and self-distillation approaches.
  • Prompt-based scaffolding: Prompt replay and curriculum re-expose hard prompts without explaining prior failures, while hint methods add directional cues that students may copy as shortcuts.The cited comparison identifies distinct limitations for replay, curriculum, and hint-based scaffolding.
  • ZPPO: ZPPO moves teacher knowledge from the gradient to the prompt through BCQ and NCQ reformulations, with responses sampled from the student’s own policy.A prompt replay buffer amplifies these reformulations inside the student’s zone of proximal development.

3. Zone of Proximal Policy Optimization

ZPPO keeps teacher-derived reasoning inside reformulated prompts rather than the policy gradient, using BCQ and NCQ to recover learning signals on hard questions while preserving response-level on-policy training. A replay buffer repeatedly exposes those questions until their mean rollout accuracy reaches 0.5 or they are FIFO-evicted.

  • 3.1 Motivation: ZPPO builds on GRPO and targets hard questions with mean rollout accuracy below 0.5.The 0.5 cutoff is used throughout because binary rewards maximize within-group standard deviation at that mean.
  • 3.2 Prompt reformulations: BCQ places one correct teacher response and one wrong student response into anonymized, shuffled candidate blocks for student discrimination.The student generates every response token from the current policy, so teacher text remains prompt context rather than a gradient target.
  • 3.2 Prompt reformulations: NCQ aggregates wrong student rollouts from the same question, explicitly listing their parsed answers and reasoning traces to expose shared failure patterns.NCQ is entirely student-generated at the response level, while its prompt structure lets the student see patterns across independently sampled failures.
  • 3.3 Prompt replay buffer: The prompt replay buffer stores only questions, admits those below 0.5 mean accuracy, and graduates them when accuracy reaches 0.5.Replayed questions remain eligible for BCQ or NCQ and therefore track the student’s current zone of proximal development.
  • 3.3 Prompt replay buffer: Replay repeatedly re-exposes hard questions with freshly sampled BCQ/NCQ candidates until graduation or FIFO eviction at capacity.Replay samples are drawn uniformly, while reformulations prioritize the hardest questions and are capped by the augmentation fraction.

4. Experiments

Across four student scales and three benchmark families, ZPPO improves macro-average performance and generalization, especially in the smallest-student regime. Ablations show that BCQ/NCQ become effective when combined with replay, while recipe choices and teacher size materially affect gains.

  • Setup: ZPPO uses Qwen3.5 students at 0.8B, 2B, 4B, and 9B with a 27B teacher, post-training each as a vision-language model.Evaluation covers 16 VLM, 10 LLM, and 5 Video benchmarks.
  • Main results: Distillation degrades LLM–Video generalization at every scale, with the sharpest losses at 0.8B, while staying within ∼1 pp of the original on VLM.The result applies to both Off-Policy and On-Policy distillation.
  • Main results: ZPPO improves the macro average across every scale and benchmark family, with the largest gains where the small-student gap is widest.The ZPPO−GRPO† gain remains robust to benchmark resampling, although the reported confidence interval measures benchmark-selection robustness rather than per-example or run-to-run uncertainty.
  • Hard-question learning: At 0% entry rollout accuracy, ZPPO graduates 28% (432/1568) of questions versus 4% (73/2035) for GRPO†; on the next-hardest, graduation is 54% versus 14%.BCQ and NCQ provide learning signals where ordinary RL has zero gradient signal.
  • Scale and teacher effects: BCQ’s contribution shrinks with scale while NCQ’s grows; larger teachers widen ZPPO’s gain, and at 0.8B the gain is most BCQ-dominated.As students scale, teacher-correct BCQ candidates dry up because remaining hard questions increasingly defeat the teacher too.
  • Component ablations: Replay alone or reformulation alone yields only modest gains, whereas pairing BCQ or NCQ with the buffer is super-additive and the full ZPPO recipe is strongest at every scale.ZPPO ties or achieves the column maximum on every benchmark in the component-ablation tables.

5. Conclusion

ZPPO keeps teacher knowledge inside prompts, preserving on-policy gradients from the student’s own rollouts while avoiding small-student generalization weaknesses from direct logit imitation and teacher prefix injection. Its structural ceiling is that BCQ cannot be formed when both teacher and student fail.

  • Conclusion: ZPPO keeps the teacher inside the prompt, allowing gradients to remain based on the student’s own rollouts.This preserves the on-policy assumption while still using teacher knowledge.
  • Conclusion: Direct logit distillation and teacher prefix-injection RL weaken generalization at small student scales, with prefix injection also breaking the on-policy assumption.Distillation imitates teacher logits, whereas prefix-injection RL splices teacher tokens into the policy gradient.
  • Conclusion: When both teacher and student fail on a question, no Binary Candidate-included Question can be constructed.This limitation creates a structural ceiling for ZPPO.

Limitations

ZPPO is limited by the teacher-bounded zone: BCQ requires teacher success, while upper-capability students approach an NCQ-only regime. Its replay of all-wrong questions also conflicts with dynamic sampling, and broader multimodal, multi-step, and agentic extensions remain outside scope.

  • Teacher-bounded zone: BCQ requires the teacher to succeed on hard questions; when both teacher and student fail, only the more limited NCQ contribution remains.At the upper end of student capability, BCQ’s contribution shrinks and the zone effectively collapses to NCQ-only.
  • Tension with dynamic sampling: ZPPO stores all-wrong questions in its replay buffer, whereas dynamic sampling deletes them, making naive combination nontrivial.A proposed fix is sequential use: first identify hard questions, then apply the methods in sequence.
  • Scope beyond reasoning correctness: ZPPO post-trains already-built compact VLMs and is orthogonal to their initial architectural compression or large-to-small distillation.The recipe could in principle be stacked after either construction route, while multi-step and agentic extensions are out of scope.

Ethical Considerations … B.1. ZPPO-77K training corpus

The paper frames ZPPO as a prompt-side alternative to logit imitation and gradient injection, while documenting inherited upstream biases and a reasoning-focused reward. Its ZPPO-77K corpus contains approximately 77K multimodal reasoning triples assembled from two public sources and filtered toward difficult, valid examples.

  • Ethical Considerations: ZPPO inherits social, demographic, linguistic, and modality biases from Qwen3.5 pretraining because the base data are neither modified nor filtered.Its reward targets benchmark answer correctness rather than safety or fairness.
  • A. Full Related Work: Knowledge distillation transfers teacher competence by training smaller students to imitate teacher logits, hidden states, or sampled outputs.The passage identifies this as the dominant transfer recipe for LLMs and VLMs.
  • A. Full Related Work: ZPPO moves teacher knowledge from the gradient to reformulated prompts, keeping every response token entering the policy gradient student-generated.BCQ and NCQ provide the prompt-side candidates on hard questions, preserving on-policy behavior at the response level.
  • A. Full Related Work: BCQ and NCQ candidates are assembled dynamically from current rollout batches, while the replay buffer amplifies reformulated prompts within the student’s current zone of proximal development.The candidates change as fresh rollouts are sampled throughout training rather than relying on pre-authored hint text.
  • B.1. ZPPO-77K training corpus: ZPPO-77K contains ∼77K (x_image, x_text, a⋆) triples, using images, text questions, and short gold answers as rule-based reward targets.The corpus is the multimodal RL training corpus used throughout the paper.
  • B.1. ZPPO-77K training corpus: The corpus combines zlab-princeton/Vero-600k’s 34 sub-datasets with OpenDataArena/MMFineReason-SFT-586K, a 586K-sample chain-of-thought VLM corpus annotated with per-example pass_rate.The sources cover STEM, chart/OCR, knowledge/recognition, counting, grounding, and search domains.
  • B. Training Dataset and Benchmarks: The corpus divides sub-datasets into two tiers according to their relevance to challenging multimodal reasoning.Tier 1 caps STEM and Chart/OCR sub-datasets at 2,800 samples each.
  • B.1. ZPPO-77K training corpus: Examples are deduplicated with a canonical-name map, retain the priority-1 Vero source, and exclude MMFineReason items with Qwen3-VL-4B-Thinking pass_rate > 0.5.Per-sample filters require answer length ≤512 characters and images at least 100 pixels in both dimensions.

B.2. Evaluation benchmarks

Evaluation uses a 31-benchmark suite spanning VLM, LLM, and Video tasks, with standardized decoding and answer extraction. Scoring combines deterministic parsing with shared or official benchmark-specific judges when exact matching is unsafe.

  • Benchmark coverage: The evaluation suite contains 31 benchmarks: 16 VLM, 10 LLM, and 5 Video benchmarks.Tables 5–7 provide abbreviations, full names, scoring protocols, and HuggingFace data sources.
  • Decoding configuration: Evaluation uses one deterministic decoding configuration across all benchmarks and model checkpoints: temperature=0.6, top-p=0.95, and top-k=20.Training rollouts instead use temperature=1.0 and top-p=1.0 to support exploration.
  • Prompt and judge templates: Upstream reasoning and answer-format directives are removed, while a shared RL closer enforces the same think/boxed answer format during training and evaluation.This aligns evaluation answer extraction with the format optimized by the policy gradient.
  • Prompt and judge templates: When deterministic parsing fails, scoring falls back to a shared LLM-as-judge, while CharXiv, MM-Vet, HLE, MultiChallenge, and MMVU use official benchmark-specific judge prompts.The judge model is the same Qwen3.5-27B-FP8 model used as the training teacher.

C. ZPPO algorithm … D.2. Baseline implementations

ZPPO keeps teacher-generated content on the prompt side while using only current-student responses in the policy gradient, combining BCQ, NCQ, replay, and zero-advantage exclusion. The implementation specifies shared training infrastructure, candidate construction and compression, and matched off-policy, on-policy, GRPO, and Hint baselines.

  • C. ZPPO algorithm: ZPPO’s policy gradient uses only responses generated by the current student; the frozen teacher supplies prompt-side BCQ candidates and compression, never gradient response tokens.This preserves the on-policy response-token invariant while allowing teacher information inside reformulated prompts.
  • C. ZPPO algorithm: The training recipe combines GRPO group-relative advantages and PPO clipping, DAPO’s asymmetric clip-higher settings and token-level aggregation, REINFORCE++’s two-step estimator, and ZPPO’s zero-advantage exclusion.DAPO uses (𝜖low, 𝜖high) = (0.20, 0.28), while ZPPO’s default computes batch statistics only over non-trivial groups.
  • C. ZPPO algorithm: BCQ presents one correct teacher trace and one wrong student trace as anonymized candidates, while NCQ aggregates the student’s wrong rollouts and parsed answers to expose shared failure modes.Both candidate types are teacher-compressed under a shared prompt and token cap to reduce obvious surface cues.
  • C. ZPPO algorithm: ZPPO replays hard questions until their mean plain-rollout accuracy reaches 𝜏=0.5, then graduates them or FIFO-evicts them when the buffer exceeds capacity.Hard questions are admitted when ¯𝑟x<𝜏, with FIFO eviction enforcing |ℬ|≤|ℬ|max.
  • D.1. ZPPO hyperparameters: 64 NVIDIA H100-80 GB GPUs run the experiments across 8 nodes, with 6 GPUs per node for student rollouts and updates and 2 for teacher services and judging.Teacher generation and candidate compression overlap with student rollout scoring rather than extending the training critical path.
  • D.1. ZPPO hyperparameters: Candidate compression preserves final answers, caps each candidate at 512 tokens, and lets NCQ include every wrong rollout within the Qwen3.5 policy’s 262K context window.Plain rollouts use a 4,096-token prompt budget, whereas BCQ and NCQ are bounded by the architectural context window.
  • D.2. Baseline implementations: All baselines share student initialization, ZPPO-77K, evaluation, rollout-side hyperparameters, and hardware, while differing on the gradient-side objective.RL methods split each rollout batch into equal-sized mini-batches; distillation methods use their respective imitation losses.
  • D.2. Baseline implementations: Off-policy distillation filters teacher trajectories for correctness and applies per-token JSD only to correct teacher targets, whereas on-policy distillation applies per-token JSD to teacher-rescored student rollouts.Both † variants add the same prompt replay buffer used by ZPPO; GRPO disables BCQ, NCQ, and replay, while GRPO† adds replay alone.

D.3. Compute cost per run … F.1. Buffer dynamics across student scales

Across compute, extended evaluations, and buffer dynamics, ZPPO’s selective teacher use keeps teacher-side cost below Off-Distill while its replay buffer fills faster for weaker students and still graduates the hardest entries at every scale. The extended 4B/9B tables preserve the main, generalization, and component-ablation evaluation structure.

  • D.3. Compute cost per run: ZPPO applies teacher-side processing only to the hardest 25% of new prompts, keeping that cost substantially below Off-Distill.Off-Distill forwards all eight student rollouts through the 27B teacher for KL targets on every prompt.
  • D.3. Compute cost per run: Student rollout compute determines ZPPO wall-clock time because teacher generation, candidate compression, and judging overlap with student rollout.The student phase is slower than the teacher phase under the shared 6:2 student/teacher hardware split.
  • E. Extended main-paper tables (4B and 9B): The 4B and 9B extensions add main-result, generalization, and component-ablation tables, with prompt-replay variants marked by †.The main and generalization tables include Δ=ZPPO−maxother, while component-ablation tables omit that row.
  • F.1. Buffer dynamics across student scales: At 0.8B and 2B, the replay buffer reaches capacity within approximately 70 and 100 steps, respectively, versus approaching capacity near step 200 at 4B and 9B.Weaker students fail more questions, causing admission to outrun graduation; every scale ends training near capacity.
  • F.1. Buffer dynamics across student scales: Graduations from the 0% admission bin rise 293 →432 →562 →514 across 0.8B/2B/4B/9B, peaking at 4B.The 0% bin corresponds to questions where ordinary RL has zero gradient signal because every rollout fails.
  • F.1. Buffer dynamics across student scales: The 0% admission bin’s graduate share increases 13% →16% →21% and remains 21% from 4B to 9B.This indicates that the hardest entries graduate across scales rather than remaining permanently trapped in the buffer.
  • F.1. Buffer dynamics across student scales: BCQ and NCQ provide usable learning signal on questions where ordinary RL stalls because every rollout fails.The buffer dynamics identify this mechanism as the explanation invoked for the main-paper results.

F.2. ZPPO vs. GRPO† training dynamics at 2B

At 2B, ZPPO’s advantage over GRPO† arises from stronger optimization signals and more successful replay-buffer outcomes. BCQ/NCQ create positive advantages for otherwise uninformative groups, increasing graduation especially among the hardest admitted questions.

  • Optimization-side dynamics: ZPPO keeps the fraction of rollouts with positive group-relative advantage ∼3 pp above GRPO† throughout training.BCQ converts an all-wrong group into a mixed group after one correct rollout, while NCQ exposes diversity after two distinct failure modes appear.
  • Replay-buffer dynamics: Under matched buffer policy, ZPPO graduates 2,627 entries versus 1,440 for GRPO†, corresponding to 26.6% versus 14.3%.The admission threshold, graduation threshold, capacity, and FIFO eviction rule are identical, but each run admits questions failed by its own plain rollouts.
  • Replay-buffer dynamics: ZPPO produces 432 versus 73 graduations from the 0% admission bin and 510 versus 169 from the 12.5% bin.These comparisons use each run’s own admitted cohort under matched buffer policy.
  • Replay-buffer dynamics: With ZPPO, the graduated fraction climbs to ∼56% by attempt 6+, while its never-improved fraction shrinks monotonically with resampling attempts.Figure 24 attributes this decomposition to ZPPO converting improved non-graduated entries into graduated entries more aggressively than GRPO†.

F.3. Multi-metric dynamics across scales

Across four student scales, ZPPO training shows rising BCQ accuracy, scale-dependent NCQ learning, and divergent entropy dynamics. BCQ improves universally, whereas NCQ remains essentially flat at 0.8B and weaker students sustain greater exploration.

  • Multi-metric dynamics: Figure 25 aggregates six per-step metrics across ZPPO runs at 0.8B, 2B, 4B, and 9B student scales.The metrics are policy-gradient loss, BCQ accuracy, NCQ accuracy, positive-advantage fraction, rollout accuracy, and actor Shannon entropy.
  • BCQ dynamics: BCQ accuracy rises from low initial values to scale-dependent convergence-regime values at every student scale.Candidate compression matches the two options’ length and format, removing obvious surface cues such as preferring the longer trace.
  • NCQ dynamics: NCQ accuracy rises at 2B, 4B, and 9B but is essentially flat at 0.8B.The 0.8B model gains little because limited capacity hinders discrimination among its own failure modes when shown collectively.
  • Entropy dynamics: ∼1.0/1.2 nats: 0.8B and 2B policies reach these entropy levels by step 200, while 4B and 9B plateau lower after a brief rise.The smaller students continue expanding their candidate distributions, whereas stronger students concentrate more quickly on near-optimal solutions.

F.4. Component ablation on the LLM and Video benchmarks

On the LLM and Video benchmarks, replay and prompt reformulation remain super-additive, while BCQ dominates at 0.8B, NCQ at 9B, and full ZPPO leads macro-averages across scale–family blocks. The ZPPO−GRPO† generalization gap is larger on LLM than VLM from 2B onward, indicating stronger gains with capacity.

  • F.4. Component ablation on the LLM and Video benchmarks: Replay × reformulation is super-additive on both LLM and Video generalization, with BCQ or NCQ plus replay exceeding the sum of isolated effects.The replay buffer re-exposes each hard question to new BCQ/NCQ reformulations.
  • F.4. Component ablation on the LLM and Video benchmarks: At 0.8B, BCQ provides the dominant generalization gain over GRPO†, whereas NCQ overtakes BCQ at 9B.The full ZPPO recipe is the column maximum on macro-average blocks for every scale and benchmark family, with limited Video exceptions for the original base.
  • F.4. Component ablation on the LLM and Video benchmarks: +3.1/+2.8/+2.6 pp on LLM versus +2.8/+2.2/+1.6 pp on VLM for ZPPO −GRPO† at 2/4/9B.From 2B onward, the buffer-isolated macro-average gap is larger on LLM than VLM.
  • F.4. Component ablation on the LLM and Video benchmarks: +4.4 pp on LLM versus +4.9 pp on VLM at 0.8B, where both gaps are large and comparable.The multimodally trained 0.8B student retains substantial headroom on both LLM and VLM benchmarks.

F.5. Hint and Prefix experimental setup and results

Hint and Prefix inject teacher-correct information into prompts or student rollouts while sharing ZPPO’s replay and training setup. Hint provides moderate VLM gains but transfers weakly beyond VLM, whereas Prefix harms LLM and Video generalization; both omit BCQ and NCQ.

  • Implementation: Hint appends a teacher-generated answer-free hint inside <hint> tags, after which the student samples a fully on-policy rollout.The teacher-derived hint augments the prompt, but response-token gradients remain on-policy.
  • Implementation: Prefix forces the first α=0.4 token-fraction of a correct teacher rollout as an anchor, masking those tokens so only student continuations enter the gradient.The student samples continuations autoregressively, following BREAD’s masked-prefix implementation.
  • Setup: Both baselines share ZPPO’s replay buffer, initialization, dataset, pipeline, and training-budget convention, but neither runs BCQ or NCQ.All other hyperparameters are identical to ZPPO.
  • Results: +1.2 pp: GRPO†+Hint improves VLM over GRPO† but remains 0.4 pp below GRPO†+BCQ and 1.6 pp below ZPPO.Hint’s one-sided guidance does not compare teacher-correct and student-wrong candidates or aggregate student wrong rollouts.
  • Results: 47.6 / 61.6: GRPO†+Hint reaches these LLM / Video scores, marginal versus GRPO†’s 47.3 / 61.9.The hint acts as a directional shortcut and supplies neither BCQ’s contrastive discrimination nor NCQ’s collective-negative avoidance.
  • Results: −1.0 pp / −0.9 pp: GRPO†+Prefix loses on LLM / Video relative to GRPO† while gaining only +0.3 pp on VLM.Prefix supplies a single positive continuation cue without paired wrong candidates or aggregated student negatives.

F.6. Benchmark-selection robustness via cluster bootstrap … G.1. ZPPO students vs. the 27B teacher

Benchmark-level cluster bootstrap shows that ZPPO’s gains are robust to benchmark selection, while candidate-match audits support genuine BCQ discrimination and reveal a scale-dependent NCQ limitation. The supplied passages do not provide evidence for the listed G or G.1 sections.

  • F.6. Benchmark-selection robustness via cluster bootstrap: ZPPO −GRPO† excludes zero in all twelve scale-family cells, with win-rate ≥8/10 everywhere and ≥9/10 except 0.8B LLM.This isolates the buffer-side ZPPO recipe while holding other RL settings fixed.
  • F.6. Benchmark-selection robustness via cluster bootstrap: Both component-isolated comparisons also exclude zero in every cell, indicating detectable macro-level contributions from both BCQ and NCQ.The comparisons are ZPPO −GRPO†+BCQ for NCQ’s marginal contribution and ZPPO −GRPO†+NCQ for BCQ’s marginal contribution.
  • F.6. Benchmark-selection robustness via cluster bootstrap: The ZPPO −Best-non-ZPPO CI excludes zero everywhere except 4B and 9B Video, where intervals are [−0.24, +0.90] pp and [−0.02, +0.86] pp.The strongest baseline there is the untrained base model, yet ZPPO still wins 3/5 and 4/5 Video benchmarks respectively.
  • F.6. Benchmark-selection robustness via cluster bootstrap: 10,000 shared benchmark resamples produce 95% CIs measuring robustness to which benchmarks enter each macro-average, not training or decoding variance.Identical comparisons receive identical intervals because each block reuses one fixed-seed benchmark resample across columns.
  • F.7. BCQ and NCQ candidate-match audit: Worked examples show both branches recovering correct answers on four of five hard questions, while the 0.8B straw example is BCQ-only because NCQ degenerates.The examples include counting, scene QA, and chart reading, with plain rollout accuracy 0/8 on the illustrated hard questions.
  • F.7. BCQ and NCQ candidate-match audit: BCQ audits favor genuine binary discrimination: accuracy is 36–69%, while correct rollouts match positive candidates 78–91% and wrong rollouts match negatives 77–97%.The anonymization and shared-compression pipeline therefore does not appear to expose a trivially separable teacher-positive surface signal.
  • F.7. BCQ and NCQ candidate-match audit: NCQ omits teacher traces entirely: 8/8 student-negative traces appear in prompts, whereas 0/4 teacher-positive traces do.Thus match-neg measures repetition of listed wrong answers rather than matching an inserted correct answer.
  • F.7. BCQ and NCQ candidate-match audit: NCQ performance is scale-dependent: accuracy is 1.2% at 0.8B and 22%/35%/31% at 2B/4B/9B, while match-neg falls from 82.7% to 2.2%/0.7%/0.2%.The pattern indicates that larger students increasingly discriminate against their own listed wrong answers, whereas the 0.8B student often repeats them.

G.2. Per-benchmark results: ZPPO students vs. the 27B teacher · G.3. Discussion

Per-benchmark PASS-AT-1 results compare four ZPPO student scales directly with the 27B teacher across LLM, VLM, and Video suites. The 9B student approaches the teacher on many benchmarks, while remaining gaps and ZPPO gains track the teacher–student capability gap and teacher saturation.

  • G.2. Per-benchmark results: ZPPO students vs. the 27B teacher: All per-benchmark values use PASS-AT-1 accuracy under a shared evaluation harness, making ZPPO student–teacher cell comparisons directly meaningful.The expanded tables cover the 27B teacher and four ZPPO student rows.
  • G.2. Per-benchmark results: ZPPO students vs. the 27B teacher: The 9B ZPPO student reaches the 27B teacher on AIME25 at 70.0 and stays within approximately 4 pp across the MMLU family without teacher logits in the gradient.The MMLU family includes MMLU, MMLU-Pro, and MMLU-Rd.
  • G.2. Per-benchmark results: ZPPO students vs. the 27B teacher: The 9B ZPPO student comes within approximately 1 pp of the teacher on OCREN, InfoVQA, MVerse, MVista, MM-Vet, and WeMath, with larger gaps on VBlind, VisP, and BabyV.On OCREN, the reported values are 56.7 for the student and 55.7 for the teacher.
  • G.2. Per-benchmark results: ZPPO students vs. the 27B teacher: At 9B, every Video benchmark lies within a few pp of the 27B teacher, with the largest residual gap on VMME at 5.9 pp.Video is a generalization evaluation even though the student is trained as a VLM.
  • G.3. Discussion: The 27B teacher varies sharply across benchmarks and remains unsaturated on HLE (16.0), OCRZH (33.5), BabyV (30.4), and OCREN (55.7).These hard math/reasoning slices sit well below the macro-averages.
  • G.3. Discussion: BCQ candidates can occur where the teacher solves questions the student cannot, but the candidate pool is thinner on hard benchmarks than on saturated ones.The passage links ZPPO’s gains to how often the teacher–student gap on a benchmark is non-trivial but non-empty.
  • G.3. Discussion: The 9B+ZPPO row is within ≤1.0 pp of the teacher on AIME25, OCREN, InfoVQA, MultiCh, MVerse, MVista, MM-Vet, and WeMath.The reported pairs include AIME25 (70.0 vs. 70.0) and OCREN (56.7 vs. 55.7); the authors do not interpret proximity as strict student superiority because run noise is comparable to residual gaps.
  • G.3. Discussion: ZPPO’s per-benchmark gains follow capability gaps that are non-trivial but non-empty, and by 9B the student has closed in on the teacher across much of the suite.The largest remaining gaps—HLE, VBlind, and VMME—align with benchmarks where the teacher itself is least saturated.
Loading 2606.18216v1…