Allspark: Weak to Strong Transfer via Alternating Chain of Thought

1 INTRODUCTION 弟子不必不如师，师不必贤于弟子。

• Large-model RL testing is expensive because repeated sampling and optimization, particularly for long reasoning traces, can bottleneck experimentation.The paper frames weak-to-strong transfer as an alternative for developing reasoning capabilities in smaller models.Source: §1 · p. 1
• Allspark trains a weak teacher alongside a frozen copy of the same model, then reuses the teacher to steer a frozen stronger student through alternating chains of thought.The strong student produces the final answer, and the shared text interface allows transfer across architectures and tokenizers.Source: §1 · p. 2
• The study examines controlled Qwen experiments across math and reasoning and larger Inkling experiments on ARC-AGI-2, including transfer within and across model families.The cross-family evaluations include Kimi K2.6 and Nemotron 3 Ultra.Source: §1 · p. 2
• The paper analyzes how teacher steering interacts with test-time scaling, identifying settings that improve accuracy and characterizing the retained-token tradeoff.Its reported comparison is accuracy against retained completion tokens.Source: §1 · p. 2

2 BACKGROUND AND RELATED WORKS

• Weak-to-strong generalization studies whether a stronger model can benefit from a weaker supervisor without inheriting its performance ceiling; prior weak-label and reasoning-trace methods update the strong model.Burns et al. (2023) finetune on weak labels, while Yang et al. (2024) use selected weak-model reasoning traces.Source: §2 · p. 2
• On-policy distillation gives teacher feedback on student-sampled sequences to reduce mismatch between training prefixes and inference prefixes, and cross-tokenizer variants provide token-level supervision across vocabularies.The passage cites Agarwal et al. (2024), Lu & Thinking Machines Lab (2025), and Niu et al. (2026).Source: §2 · p. 2
• Direct-OPD, W2S-OPD, and OPRD transfer weak-model policy changes using strong-student rollouts and update strong-student parameters, unlike applying a trained weak teacher to frozen students.The cited methods are Direct-OPD (Feng et al., 2026), W2S-OPD, and OPRD (Park et al., 2026).Source: §2 · p. 2
• Other approaches guide a frozen strong model at inference using weak-model likelihood differences, a trained weak collaborator, learned deferral, or weak-critic feedback.Examples include weak-to-strong search, CoWeST, Co-LLM, and weak critics.Source: §2 · p. 3

3 ALLSPARK: TRANSFER WITHOUT STRONG-MODEL TRAINING ROLLOUTS

• Allspark trains the weak teacher entirely with weak-model rollouts, then replaces the frozen weak student with a frozen strong student at inference without updating the strong model.The strong student continues the shared reasoning and produces the final answer.Source: §3 · p. 3, Figure 1 · p. 3

3.1 ALTERNATING REASONING

• The teacher and student take turns extending a shared reasoning prefix, and the student generates the final answer after reasoning stops.The prefix begins with the problem, and each generated chunk is appended before the next turn.Source: §3.1 · p. 3

3.2 TRAINING THE WEAK TEACHER

• During training, the student is a frozen copy of the teacher’s initial model, while only the teacher is optimized; direct loss is applied to sampled teacher reasoning tokens, not student tokens or boundary markers.A teacher chunk’s continuation value comes from the student’s subsequent rollout and final reward, so the chunk need not solve the problem by itself.Source: §3.2 · pp. 3–4

3.3 TRANSFER ACROSS MODEL SIZE AND FAMILY

• At transfer, both models are fixed and the stronger student replaces the frozen training partner, continuing reasoning from teacher-generated text without requiring a shared vocabulary.Each model renders and tokenizes the shared text through its own interface.Source: §3.3 · p. 4

4.1 CONTROLLED QWEN EXPERIMENTS

• With the 4B student, Allspark-trained guidance was approximately neutral in math and positive in reasoning, while untrained or ordinary-RL teachers lowered accuracy in both domains.The controlled comparisons use frozen students and a Qwen3-1.7B teacher; Table 2 reports development accuracy for 128 questions per domain.Source: §4.1 · pp. 4–5, Table 2 · p. 5
• Selected traces show M2’s shared correct solution and R5’s B-to-C change; M3 recovers to 567 and R1/R2 change to correct, though the R2 teacher overstates its passage.The P1 alternating final gives the correct answer, 1, but both P1 finals hit the answer-token limit.Source: §4.1 · p. 5, Figure 2 · p. 6, Figure 3 · p. 7

4.2 INKLING EXPERIMENTS

• 81.6% was Allspark’s highest observed accuracy versus 78.1% for Inkling alone; at a selected point, it reached 79.2% with 10.8k tokens versus 78.1% at 12.3k.Paired runs use the rl’ed teacher with Inkling-Small or Inkling across four thinking efforts on 96 ARC development tasks.Source: §4.2 · pp. 5, 7, Figure 4 · p. 7

4.3 CROSS-FAMILY TRANSFER

• Kimi accuracy rises by 11.5 percentage points with 7.7% fewer retained tokens; Nemotron accuracy rises by 17.7 and 5.2 percentage points in medium- and full-thinking modes, with increased token use.The same Inkling-Small teacher is reused with frozen cross-family students; handoffs exchange reasoning as text across different tokenizers.Source: Figure 5 · p. 8, §4.3 · p. 8

• Allspark trains a weak teacher alongside a frozen copy, then uses the teacher to steer a stronger frozen student at inference.The shared text interface supports transfer across model sizes and families without strong-model rollouts during training.Source: §5 · p. 8
• The study spans Qwen math and reasoning and larger-model ARC-AGI-2 experiments, with Inkling results demonstrating cross-family transfer and improved accuracy–token tradeoffs.Source: §5 · p. 8
• The authors suggest using smaller models to explore RL recipes and training data, then reusing the resulting teachers with stronger students.They also propose extensions to model alignment and AI safety research.Source: §5 · p. 8

6 LIMITATIONS

• Inference serves both the teacher and student, and extra calls and handoffs may increase cost and latency.Fewer retained tokens therefore do not necessarily mean lower deployment cost.Source: §6 · p. 9
• Evidence comes from a restricted set of models, tasks, and training runs, so broader studies are needed to establish robustness and generality.Source: §6 · p. 9
• The implementation depends on interrupting and resuming a shared reasoning stream; use with non-thinking models or interfaces without such a stream is untested.Source: §6 · p. 9

AI USE STATEMENT

• Generative AI was used to polish writing, draft experiment settings, and discover related work.Source: AI USE STATEMENT · p. 9
• The authors report not using generative AI for research ideation, synthetic-data generation, core analysis, or proving mathematical claims.Source: AI USE STATEMENT · p. 9
• They also used AI to create or modify scientific figures and images and edit code, reviewed AI-assisted work, and accept responsibility for the final content.Source: AI USE STATEMENT · p. 9

A.1 MODELS AND ROLLOUT MASKS

• Both arms start from Qwen3-1.7B with seed 0; only LoRA parameters are optimized, and Allspark’s student has no adapter.Source: §A.1 · p. 12
• Allspark begins with up to 128 student reasoning tokens, then alternates chunks of up to 256 tokens until reasoning ends or reaches the shared ceiling.Each writer conditions on the prompt and complete preceding stream; the frozen student generates the final answer.Source: §A.1 · p. 12
• A binary verifier reward for the final answer trains teacher reasoning tokens; student and forced closing tokens have zero loss mask.There is no intermediate reasoning reward.Source: §A.1 · p. 12

A.2 RL LOSS: MASKED SAMPLER-HEAD CENTERING

• Allspark applies the shared RL loss only to teacher reasoning tokens, whereas ordinary RL applies it to the adapted model’s reasoning and answer tokens.Source: §A.2 · p. 12
• Group-standardized binary answer rewards give each trajectory one advantage, reused at every trainable token; zero reward variance yields zero advantage.The group uses G = 32 trajectories, with no epsilon added to the standard deviation.Source: §A.2 · p. 12
• The token loss scores sampled tokens relative to a weighted center over the saved sampler’s top-K probabilities, omitting the sampled-token term when it falls outside the head.The head uses K = 128, retains its original probability mass, and omits the tail rather than modeling it.Source: §A.2 · pp. 12–13

A.3 ASYNCHRONOUS RL AND FILTERING

• Three rollout actors feed a learner that updates from six accepted prompt groups, while each group’s sampler version may lag by at most two updates.Older queued groups are discarded; the optimization uses no importance-sampling ratio or PPO clipping.Source: §A.3 · p. 13
• Prompts are screened for 1–7 successes in an eight-trajectory group, then retained training groups require 1–31 successes among 32 fresh trajectories.Each batch uses six distinct prompts, with two per domain.Source: §A.3 · p. 13

A.4 SOLUTION-LENGTH CONTROLS AND TERMINATION

• Reasoning and answer have separate hard generation ceilings, and Allspark shares the reasoning ceiling across both writers’ chunks.All returned raw tokens remain in the shared stream, without trimming or prefix cropping.Source: §A.4 · p. 13
• A reasoning-cap hit does not itself force zero reward: the saved final answer is still scored by the verifier.The loss sums trainable-token terms without per-sequence length normalization, and the longer evaluation protocol cannot repair truncated training rewards.Source: §A.4 · p. 14

A.5 OPTIMIZATION SETTINGS

• Gradients accumulate over the accepted batch before one clipped optimizer step; loss arithmetic uses FP32 and the base model is loaded in BF16.The implementation also uses gradient checkpointing and computes vocabulary projections in chunks of 32 loss-bearing positions.Source: §A.5 · p. 14

A.6 DEVELOPMENT EVALUATION AND CHECKPOINT SELECTION

• The frozen panel had 384 questions—128 each from MATH, TACO, and LogiQA 2.0—and each candidate answered through the same unadapted Qwen3-1.7B student.There was one sampled answer per question, with seed 0; the student wrote the final answer.Source: §A.6 · p. 14
• Exact ties favored the earlier update, while reasoning accuracy and the three-domain mean were reported separately and did not determine checkpoint selection.Source: §A.6 · p. 14
• The completed review selected Allspark checkpoint 94; ordinary-RL checkpoint 92 retained its earlier nomination.Table 4 reports correct answers on two 128-question development domains; these are development results, not public-test scores.Source: §A.6 · p. 15, Table 4 · p. 15

A.7 DEVELOPMENT LEARNING CURVES

• Figure 6 tracks checkpoints 0, 10, 20, 50, and 100 on 128 fixed questions per domain, reporting math and reasoning only.The historical 8K protocol allowed 4,096 reasoning tokens and 1,536 answer tokens.Source: Figure 6 · p. 15

A.8 TRAINING DIAGNOSTICS

• The plotted pass rates use accepted, filtered training rollouts, and overall is the equal-domain mean; learner entropy is measured on each method’s trainable-token contexts.Source: Figure 7 · p. 16
• Because accepted prompts and rollout policies differ between methods, the one-seed curves are descriptive diagnostics, not held-out accuracy or paired treatment effects.Source: Figure 7 · p. 16
• Figure 7 covers the common first 100 updates, although ordinary RL ran to 110 and Allspark to 100; update counts do not equal tokens or GPU time.Screening and rejected rollouts also contribute to training cost.Source: §A.8 · p. 16, Figure 7 · p. 16

B.1 INKLING TRAINING AND INFERENCE

• A rank-32 LoRA adapter trained on Inkling-Small for 72 rollout steps alongside a frozen copy that opened each trace and generated the visible answer.Only teacher-generated reasoning positions received policy-gradient loss; the length penalty was zero for steps 0–35, then increased from step 36, with reasoning capped at 24,576 tokens.Source: §B.1 · p. 17

B.2 CROSS-FAMILY INFERENCE PROTOCOL

• Cross-family evaluation alternated a frozen Inkling-Small teacher with frozen Nemotron or Kimi students through shared text re-encoded with each receiver’s tokenizer, with the student starting and answering.Token IDs were not passed between model families, and equal numerical effort or reasoning compute across families was not assumed.Source: §B.2 · pp. 17–18

B.3 EARLIER WITHIN-FAMILY CONTROL

• The earlier within-family control evaluated frozen Inkling-Small and Inkling students alone, with an untrained teacher, and with the checkpoint-35 Allspark teacher at effort 0.7 on 96 development tasks.Each condition used one attempt per task; all model weights remained fixed during evaluation.Source: §B.3 · p. 18, Table 6 · p. 18

B.4 DOLLAR COST AND LATENCY

• For Kimi, Allspark reduced retained tokens 7.7% but increased modeled cost 21.1%, from approximately $0.1466 to $0.1774 per attempt.At the same Inkling effort, Allspark cost 1.42–1.82× the student-alone baseline; a different-effort Inkling comparison used 12.1% fewer retained tokens (10,771 versus 12,253) but only about 2.7% lower modeled cost ($0.0606 versus $0.0623).Source: §B.4 · p. 19

C ROLLOUT CASE STUDIES

• Across seven selected paired rollouts, four incorrect answers became correct, two correct answers stayed correct, and one correct answer became incorrect.These cases illustrate exchanges rather than establishing the causal effect of any individual teacher span.Source: C ROLLOUT CASE STUDIES · p. 20

READING THE EXCHANGE

• The student adopts the teacher’s even-total invariant, and the final code rejects the one-node case.The exchange centers on splitting a tree into even-sized components.Source: READING THE EXCHANGE · p. 26