PaperlayerView exact revision on arXiv

Source-linked AI summary

Accordion: Elastic-Depth Inference via Fixed-Prefix Layer Looping

Yudong Zhang, Wei Ding, Kaili Zheng, Yuchen Ding, Zhenyu Zhang

arXiv:2610.32492v1cs.LGcs.CL

TL;DR

Language-model applications need varied inference budgets, while maintaining separate models is costly and jointly optimized looped models can compromise shallow-path performance. Accordion uses fixed-prefix layer looping to provide five budgets in one model and preserves them through supervised fine-tuning and reinforcement learning. Increasing depth raises the base model’s five-category average across 37 tasks by 3.01 percentage points, and the final model surpasses each domain expert on its task-and-budget average.

  • Problem

    Language-model applications need varied inference budgets, but separate models are costly and jointly optimized looped models can compromise shallow-path performance.

    Source: §1 · p. 2

  • Method

    Accordion provides five fixed-prefix layer-looping paths in one MoE Transformer and preserves the budgets through SFT and RL with expert merging and budget-matched distillation.

    Source: §1 · p. 2

  • Results

    3.01 percentage points: increasing depth raises Accordion-6B-A0.5B’s five-category average across 37 tasks, and Accordion-RL surpasses each domain expert on its corresponding task-and-budget average.

    Source: §1 · p. 2, §4.2.4 · p. 21

  • Takeaways & Limitations

    A single model retains depth-dependent gains through post-training, including in mathematical reasoning and code generation.

    Source: §7 · p. 24

    The Diverse-QA test sets retain questions flagged by a source-level overlap audit, so reported scores include potential benchmark contamination.

    Source: §3.3 · p. 11

Abstract

from arXiv · show

Large language models serve applications with diverse inference budgets, but training and maintaining separate models for each budget incurs substantial costs. Recent Looped Transformers enable flexible inference computation through parameter reuse, but maintaining strong performance across different inference budgets remains an open challenge. We introduce Accordion, a mixture-of-experts-based model that supports five inference budgets through fixed-prefix layer looping. Each budget follows a pre-defined execution path, while the minimum budget retains the unlooped model. Joint pretraining samples one budget per micro-batch and optimizes its next-token prediction loss without additional cross-budget consistency objectives. We further preserve these inference budgets through supervised fine-tuning and reinforcement learning, where parameter-group merging and budget-matched multi-teacher on-policy distillation integrate three domain experts into one model. After 3.6T tokens of pretraining, increasing inference depth improves Accordion-6B-A0.5B's five-category macro-average across 37 tasks by 3.01 percentage points, with the middle budget capturing 89% of this gain using half as many additional layer executions. The final model surpasses each domain expert on its corresponding task-and-budget average, while achieving strong instruction-following performance across all budgets and retaining depth-dependent gains in mathematical reasoning and code generation.

1 Introduction

Accordion addresses the challenge of supporting varied inference budgets without maintaining separate models, using fixed-prefix layer looping in a shared MoE Transformer. The paper reports depth-dependent gains from pretraining through post-training, including expert-level comparisons and instruction-following results.

1 Introduction

  • The work targets the cost of training and maintaining separate models for different inference budgets, a challenge that remains in jointly optimized looped models.The introduction also notes that elastic inference must remain effective through supervised fine-tuning and reinforcement learning.Source: §1 · p. 2
  • Accordion uses five fixed inference paths in one MoE Transformer, repeating a selected layer prefix while retaining the unlooped path at minimum budget.Training samples one budget per micro-batch and optimizes its next-token prediction loss without a cross-budget consistency objective; inference fixes one budget per request.Source: §1 · p. 2, Figure 2 · p. 3
  • SFT preserves the five budgets, while RL integrates three domain experts through parameter-group merging and budget-matched multi-teacher on-policy distillation.The experts address instruction following, mathematical reasoning, and code generation.Source: §1 · p. 2
  • 3.01 percentage points: increasing inference depth raises Accordion-6B-A0.5B’s five-category average across 37 tasks, improving 36 of 37 benchmark scores.The middle budget captures 89% of this gain with half as many additional layer executions.Source: §1 · p. 2
  • Under equal training tokens, Accordion’s validation losses are within 0.003 nats of budget-specific specialists at three evaluated budgets, including the unlooped path.Fixed-prefix layer selection also achieves the lowest observed validation loss at all four looped budgets among eight evaluated strategies.Source: §1 · p. 2
  • Accordion-RL surpasses each domain expert on its corresponding task-and-budget average, while the final model outperforms four public references on IFEval across all five budgets.Additional depth also improves mathematical reasoning and code generation, though the optimal budget varies across tasks.Source: §1 · p. 2

2 Accordion

Accordion provides five fixed, shared-weight execution paths by repeating prefixes of a middle layer window. Design studies select fixed-prefix looping, five budgets, shared routing, and budget-independent parameters; the section also compares the shared model with budget-specific specialists and traces its use through SFT and RL.

2.1 Architecture

  • Each request selects one of five static paths sharing weights; selected layers repeat individually, while zero-initialized gates make the second pass an identity at initialization.Source: §2.1 · p. 4

2.2 Designing Strong Shared Paths

  • Design studies use a 1.61B-parameter MoE, 80B training tokens, and held-out web validation to compare path-sharing and routing choices.Source: §2.2 · p. 5

2.2.1 Layer Selection

  • Fixed prefix (7–18) has the lowest observed validation loss at all four looped budgets and the five-budget mean, leading alternatives by approximately 0.0025–0.0041 nats.Source: §2.2.1 · p. 5, Figure 3 · p. 6

2.2.2 Number of Training Budgets

  • At equal expected training computation, five budgets yield the lowest observed loss at k = 0 and 12 and lower loss at k = 6 than the three-budget model.The four-budget model was not trained at k = 6.Source: Table 1 · p. 6, §2.2.2 · p. 6

2.2.3 Router Sharing

  • The shared router achieves the lowest five-budget mean loss, improving on separate routing by 0.0013 nats and on separate routing with exclusion by 0.0017 nats.Source: Table 2 · p. 6, §2.2.3 · p. 6

2.2.4 Budget-Conditioned Parameters

  • Shared gates and normalization parameters achieve the lowest observed loss at every evaluated budget, so Accordion retains budget-independent parameters.Source: Table 3 · p. 7, §2.2.4 · p. 7

2.2.5 Comparison with Budget-Specific Specialists

  • Within 0.0030 nats of specialists at all three budgets under equal tokens, Accordion improves on the FLOPs-matched specialist by 0.0068 nats at the largest budget.The equal-token comparison uses 80B tokens; the FLOPs-matched comparison adjusts specialist token counts to match modeled training computation.Source: Figure 4 · p. 7, §2.2.5 · pp. 7–8

2.3 Training Pipeline

  • After 3.6T tokens at 8K, 50B further tokens extend the model to 32K; SFT and RL retain five paths, with RL merging three experts through budget-matched distillation.Source: Figure 5 · p. 8, §2.3 · p. 8

3 Pretraining

Accordion-Base uses a 3.6T-token pretraining curriculum and five inference budgets to turn added layer executions into broad capability gains. The middle budget captures most of the aggregate improvement, while longer contexts can benefit from deeper paths.

3.1 Data

  • The curriculum samples 3.6T tokens from a pool of approximately 4.68T tokens across web text, mathematics, code, and reasoning sources.It progressively increases mathematics, code, and reasoning exposure, then adds 50B tokens for 32K context extension.Source: §3.1 · pp. 8–9, Figure 6 · p. 9

3.2 Hyperparameters

  • Accordion-6B-A0.5B’s 3.6T-token, 8K curriculum trains the same five paths with a single-trajectory objective, uniform micro-batch budget sampling, and stage-specific schedules.The context-extension run starts from the annealing weights and resets the optimizer and step counter.Source: §3.2 · pp. 9–10, Table 5 · p. 10

3.3 Evaluation

  • Evaluation uses the same 37 tasks, configurations, and examples across budgets, with an aggregate that weights five task categories equally.Training trajectories and paired uncertainty estimates use the original 18-task core suite.Source: §3.3 · pp. 10–11

3.3.1 General Capabilities and Model Comparison

  • Across all five budgets, Accordion-Base beats four evaluated references on full MATH and ASDiv, with peak margins of 3.5 and 4.8 percentage points.Increasing k from 0 to 12 improves all five category averages; the largest gains are in code, reasoning, and mathematics.Source: §3.3.1 · p. 12, Figure 7 · p. 12

3.3.2 Effect of the Inference Budget

  • At k = 6, Accordion retains 89% of the endpoint quality gain while time per output token rises approximately 21–24% across measured workloads.The five budgets offer distinct operating points, and the best budget varies by task; for example, HumanEval and GSM8K peak at k = 9, while MBPP+ peaks at k = 12.Source: §3.3.2 · p. 13, Figure 8 · p. 13

3.3.3 How Budget Gains Develop during Training

  • All four looped budgets outperform k = 0 at every evaluated checkpoint, and the 18-task core endpoint advantage grows from 1.53 points early in training to 3.74 points at annealing’s end.Annealing improves all paths while preserving the benefit of additional inference computation.Source: §3.3.3 · p. 13, Figure 10 · p. 14

3.3.2 Effect of the Inference Budget

  • 47.05 to 50.06 (+3.01 percentage points): increasing k from 0 to 12 improves the 37-task aggregate and 36 of 37 task scores.The middle budget k = 6 retains 89% of the endpoint gain.Source: §3.3.2 · pp. 12–13, Figure 9 · p. 14

3.3.4 Long-Context Performance

  • On Accordion-Base-32K, the RULER mean gain from k = 0 to k = 12 rises from 0.72 points at 4K to 5.47 points at 32K context.At 16K, k = 3 captures nearly all of the endpoint gain; at 32K, gains continue through k = 9.Source: §3.3.4 · p. 13, Figure 11 · p. 15

4 Post-Training

Post-training preserves Accordion’s five inference budgets while adding conversational reasoning, instruction following, and domain-specific capabilities. The final student consolidates three experts and outperforms each on its corresponding task-and-budget average.

4.1.1 Data

  • SFT uses 71.7B input tokens, of which 68.5B contribute to masked training loss, with one epoch for mathematics and code and two for dialogue, instruction-following, Exercism/SQL, and safety.The training set is filtered for overlap with GSM8K, full MATH, HumanEval, MBPP, and AIME 2024.Source: §4.1.1 · p. 15, Figure 12 · p. 16

4.1.2 Hyperparameters

  • Accordion-Instruct starts from Accordion-Base-32K and trains all five paths using uniform micro-batch budget sampling.SFT uses 256 packed 32K-token sequences per global batch and runs for 8,614 steps.Source: §4 · p. 15, §4.1 · p. 15, §4.1.2 · p. 16

4.1.3 Evaluation

  • Additional depth improves Accordion-Instruct’s mathematical reasoning and code generation, with the largest gains on competition mathematics and LiveCodeBench, under the sampled conversational protocol.It exceeds Qwen3-1.7B on HumanEval+ at k = 12, MBPP+ at k = 6, and AIME 2025 at k = 9; the HumanEval+ margin is 2.44 percentage points. Models use native chat templates with common sampling and scoring settings.Source: §4.1.3 · p. 16

4.2.2 Hyperparameters

  • The RL pipeline trains instruction-following, mathematics, and code experts, merges their parameter updates, then uses multi-teacher on-policy distillation to train one student.For each response, the teacher is selected by prompt domain and evaluated at the student’s loop budget; the student generates its own responses.Source: §4.2 · p. 17, §4.2.2 · p. 18, §4.2.1 · p. 18
  • Expert training raises held-out domain validation scores, while token-level distillation reduces teacher–student discrepancy and raises monitored task reward.The final student is selected after ten distillation steps using downstream comparisons with later steps.Source: Figure 14 · p. 19, §4.2.2 · p. 19

4.2.3 Evaluation

  • Accordion-RL exceeds all four evaluated public references on IFEval at every budget, by at least 16.64 percentage points over the strongest reference.All 15 matched RL-versus-SFT comparisons on IFEval, IFBench, and MMLU-Redux show higher scores after RL.Source: §4.2.3 · p. 19, Table 9 · p. 20, Figure 15 · p. 20
  • Increasing Accordion-RL’s budget from k = 0 to k = 12 improves all four mathematics tasks and all three code tasks.Task-specific peaks include LiveCodeBench at k = 6, IFBench at k = 9, and MATH500 at k = 12.Source: §4.2.3 · pp. 19–20

4.2.4 Discussion

  • Accordion-RL surpasses each of its three domain experts on its corresponding domain average across budgets, with paired 95% intervals excluding zero.Grouped merging and budget-matched distillation contribute complementary gains; distillation adds 2.30 points to the mathematics average over the grouped initialization.Source: Figure 16 · p. 21, §4.2.4 · p. 21

5 What Changes in the Second Pass?

Accordion’s repeated passes select experts and write updates that differ from the first pass, but the effects of removing repetitions do not simply add together.

5.1 Shared Weights Support Input-Dependent Expert Selection

  • At layer 9, the second pass retains an average of only 1.52 of its eight routed experts; most tokens share at most one expert between passes.Changing the input reproduces actual expert overlap to within 0.20 experts, while changing balancing bias alone largely preserves selection.Source: §5.1 · p. 21, Figure 17 · p. 22

5.2 Repeated Layers Write Distinct Updates

  • In layers 7–12, gated second-pass MoE writes remain comparable in size to first-pass writes but are nearly orthogonal, while residual-stream cosines stay above 0.967.Even tokens routed to the same eight experts across passes receive distinct outputs; second-pass write removal also increases NLL after SFT and RL.Source: Figure 18 · p. 22, §5.2 · p. 22

5.3 The Effects of Repeated Layers Do Not Simply Add Up

  • For adjacent triplets, joint removal costs 15–33% of summed individual-removal effects, showing those effects cannot be treated as independent contributions.For layers 7–9, individual NLL increases sum to 24.65 millinats/token, while joint removal increases NLL by 8.07; endpoint gains must therefore be assessed on the complete budget path.Source: Table 10 · p. 23, §5.3 · p. 23

6 Related Work

Accordion builds on layer looping, which reuses layers locally before advancing through the stack. Its fixed-prefix paths address the challenge of preserving shallow-path quality while benefiting from deeper computation.

6.1 Looped Transformers

  • Layer looping reuses each layer locally before advancing, unlike stack or block recurrence, which passes representations through multiple layers per iteration.Accordion assigns each budget a fixed prefix of a middle window while retaining the unlooped path.Source: §6.1 · p. 23

6.2 Elastic Inference

  • Accordion trains fixed-prefix paths by uniformly sampling one path per pretraining micro-batch, while callers select one fixed path for every token in a request.This path-design approach addresses the challenge of preserving shallow-path quality while benefiting from deeper computation.Source: §6.2 · p. 24

7 Conclusion

Accordion makes inference depth selectable within one model through fixed-prefix layer looping. The evaluation distinguishes its 8K base model, trained on 3.6T tokens, from the 32K context-extended model.

7 Conclusion

  • Accordion’s conclusion combines five selectable inference paths with depth-dependent gains and domain-expert integration in one set of weights.The model also outperforms the four evaluated public references on IFEval at every budget and retains gains in mathematical reasoning and code generation.Source: §7 · p. 24

A Comparison of Execution Structures

B Pretraining Data and Evaluation Details

  • Accordion-Base is the 8K model trained on 3.6T tokens; Accordion-Base-32K adds 50B context-extension tokens and initializes SFT.Both have results on the same 37 general-capability benchmarks at all five budgets; the main benchmark comparison reports Accordion-Base.Source: B Pretraining Data and Evaluation… · p. 31

B.1 Data Accounting

The appendix documents data accounting and evaluation protocols, then reports depth gains on held-out language modeling and the core benchmark suite. It also specifies the model-design study and the 32K evaluation scope.

B.1 Data Accounting

  • The 3.6T-token run samples selected shards from approximately 9.57T-token source pools, reserving one shard each from three sources for validation.Selected shards contain approximately 4.68T tokens; the separate context-extension mixture allocates 25B tokens each to long documents and replay. The source-pool overlap audit flags 73 of 1,319 GSM8K questions and 3 of 500 MATH500 questions, but actual training-stream exposure depends on sampling.Source: §B.1 · pp. 31–32, Table 11 · p. 32, Table 12 · p. 32

B.2 Evaluation Protocol and Aggregation

  • The evaluation defines core, secondary, and full-table aggregates using fixed category scopes, and reports rankings as descriptive rather than statistical-significance tests.The 37-task overall score averages five category means; benchmark scores include flagged questions in the full evaluation sets.Source: §B.2 · pp. 32–33, Table 14 · p. 33

B.3 Held-Out Language-Modeling Loss

  • Increasing inference depth monotonically reduces held-out loss in web text, mathematics, and code on identical validation tokens.The losses are for Accordion-Base before context extension; sequence-level records for paired intervals are unavailable.Source: §B.3 · pp. 33–34, Table 15 · p. 34, Table 16 · p. 34

B.4 Common-Protocol Model Specifications

  • The common-protocol comparison includes Accordion-Base, Qwen3-0.6B/1.7B, OLMoE-0125, and Ouro-1.4B.OLMoE combines approximately 5.1T training tokens; Ouro uses four recurrent steps with early exit at step three. Active-parameter counts describe an unlooped pass and exclude later SFT/RL exposure.Source: §B.4 · p. 34, Table 17 · p. 34

B.5 Paired Evidence for Depth Gains

  • The core aggregate rises from 52.49 at k = 0 to 56.23 at k = 12 (+3.74 points; paired 95% interval [3.17, 4.34]).All 18 core-task point estimates improve; paired intervals exclude zero for 12 tasks, and k = 6 retains 83% of the endpoint gain descriptively.Source: §B.5 · p. 34, Table 18 · p. 35

B.6 Model-Design Study Configuration

  • The model-design studies use a 1.61B-parameter configuration trained on 80B tokens, distinct from the main Accordion model.The configuration table defines active parameters for a single unlooped forward pass, excluding the input embedding and output head.Source: §B.6 · p. 35, Table 19 · p. 35

B.7 Accordion-Base-32K Evaluation

  • Accordion-Base-32K retains all five budgets, with complete results across 185 task scores and held-out losses on the same 8K sequences used for Accordion-Base.Its category summary averages the displayed scores, including the Ouro correction.Source: §B.7 · p. 35, Table 20 · p. 35, Table 21 · p. 36

B.7.1 Long-Context Evaluation Protocol

Long-context evaluation pairs budgets on identical examples across five inference paths and four context lengths. Internal probes examine repeated-layer computations, including expert outputs, attention readouts, and their sensitivity to interventions.

B.7.1 Long-Context Evaluation Protocol

  • The RULER comparison uses 13 tasks, greedy generation, and 500 examples per task and context length, pairing budget contrasts on identical examples.Context and output together can exceed nominal sequence length by approximately 50 tokens; the cross-implementation check is limited to three reported 4K tasks.Source: §B.7.1 · pp. 37–38
  • Endpoint-gain intervals exclude zero at all four context lengths; at 32K, gains from k = 3 to k = 9 concentrate in variable tracking and multi-key retrieval.RULER averages 13 tasks with 500 examples per task and context length; the reported endpoint gain compares k = 0 with k = 12.Source: §B.7.1 · p. 37, Table 23 · p. 37

C.1 Probe Protocol and Internal Representations

  • The probes edit second-pass writes while preserving first-pass execution, then compare teacher-forced NLL with the intact k = 12 baseline.Removing all second-pass writes recovers k = 0; suffix removals recover intermediate budgets, and the probes measure inference-time sensitivity within one training run.Source: §C.1 · p. 38
  • Even with identical routed expert sets, second-pass routed-expert output differs from first-pass output by relative norms of 0.39–0.87 across layers.Inputs and mixture weights can change despite identical expert sets; mean gate magnitudes and second-pass exposures also covary with layer position.Source: §C.1 · p. 39
  • Queries and keys retain cosines of 0.92–0.98 across passes, while output-projection cosines range from −0.07 to 0.88 and attention mass on the initial EOD token rises by 0.137.The same head retains 72% of its top-eight targets, compared with 42% for another KV group and 53% for the same KV group.Source: §C.1 · p. 39

C.2 Write Interventions and Persistence through Post-Training

The section probes how second-pass writes affect prediction, how their effects interact across layers, and how depth gains vary with input structure. It also describes SFT training and evaluations under different decoding policies.

C.2 Write Interventions and Persistence through Post-Training

  • 14.25, 15.26 and 10.63 millinats/token are the costs of removing all second-pass writes on web text, mathematics and code, respectively.Across Base-32K, Instruct and RL, removal raises NLL by 13.38, 9.66 and 7.22 millinats/token, respectively, with positive 95% intervals at each stage.Source: §C.2 · p. 39, Table 27 · p. 40

C.3 Layer-Group Structure of Nonadditive Effects

  • Contiguous six- and nine-layer removals have joint/sum ratios of 0.16–0.27, while the two spaced six-layer sets yield 0.84 and 0.93.Adjacent triplet ratio intervals fall below one; the 10+11 pair has a 0.47 ratio, versus 0.97 for 9+10.Source: §C.3 · p. 41, Table 28 · p. 41, Table 29 · p. 41
  • The tested adjacent and spaced groups contain different layers, and many internal removals produce paths outside the five trained prefixes.These interventions characterize sensitivity but do not establish why fixed-prefix training outperforms other path designs.Source: §C.3 · p. 41

C.4 Length Dependence and Structured Prediction

  • At k = 12, the long/short gain ratio rises from 0.85 at 2.01T tokens to 1.14 at 3T and 1.44 for Accordion-Base-32K.Across checkpoints, longer-bin gains remain within 13.2–15.1 millinats/token, while the shortest-bin gain falls from 20.0 to 4.5.Source: §C.4 · p. 42, Table 30 · p. 42, Figure 21 · p. 42
  • With 32 demonstrations, teacher-forced exact match rises from 8.5% to 12.0% (+3.5 points), and answer NLL falls by 0.259 nats at k = 12.The exact-match gain interval is [1.25, 5.75] points; at positive demonstration counts below 32, paired gain intervals include zero.Source: §C.4 · p. 42, Table 31 · p. 42

D Supervised Fine-Tuning Details

  • Accordion-Instruct continues training Accordion-Base-32K for 8,614 steps, with all five inference budgets sharing the resulting weights.The common conversational protocol supports paired comparisons with Accordion-RL at each budget.Source: D Supervised Fine-Tuning Details · p. 43

D.1 Data Sources and Packing

  • The SFT dataset contains 71.734B input tokens, of which 68.5B contribute to the loss, across 12 packed training components.Table 32 separates mathematics, code, and instruction/other source counts.Source: §D.1 · p. 43, Table 32 · p. 43

D.2 Optimization

  • SFT resets the optimizer and iteration counter, and uses batches of 256 packed 32,768-token sequences; paired evaluation controls prompts, limits, and scorers across SFT and RL.Depth gains persist under greedy and sampled decoding: all eight non-AIME tasks improve from k = 0 to k = 12 with greedy decoding, and seven improve with sampling; sampled LiveCodeBench improves monotonically, while sampled IFEval peaks at k = 6.Source: §D.2 · p. 43, §D.3 · pp. 43–44, Table 34 · p. 44

D.4 Common-Protocol Public Post-Training Comparison

The common-protocol comparison evaluates public reference models and reports SFT scores alongside completion and accuracy diagnostics. It also describes the public collections used to source RL data.

E.1 Data Sources

  • The comparison includes Qwen3-0.6B, Qwen3-1.7B, OLMoE-1B-7B-0125-SFT and Ouro-1.4B-Thinking, while RL data come from public instruction-following, mathematics and code collections.Reported source counts precede difficulty and code-verifier filtering.Source: §D.4 · p. 44, §E.1 · p. 45, Table 35 · p. 45

D.4 Common-Protocol Public Post-Training Comparison

  • On MATH500, sampling raises k = 0 accuracy from 69.4% to 83.4%, and added depth raises sampled accuracy to 89.2%.The sampled k = 0 to k = 12 gain is 5.80 points, versus 12.80 points under greedy decoding.Source: §D.4 · pp. 44–45
  • On sampled LiveCodeBench, the fraction of responses without a closing reasoning tag falls from 36.8% to 21.1% as k increases from 0 to 12.Budget gains reflect both answer quality and whether the reasoning trace reaches a final answer.Source: §D.4 · pp. 44–45

E.2 Expert Training and Model Selection

Three domain experts are trained from Accordion-Instruct and selected for integration into a student. Distillation signals improve over ten steps, while checkpoint comparisons are conditional on the selected models.

Expert training and model selection

  • The three domain experts share Accordion-Instruct initialization and are selected using validation across five budgets, with entropy thresholds stopping training.Training prompts and budget assignment follow the expert-training setup; the mathematics checkpoint remains at step 30 because the step-40 increase is below its paired standard error.Source: §E.2 · pp. 45–46, Table 36 · p. 46
  • The selected student is MOPD step 10, and reported paired intervals quantify evaluation-sample uncertainty without adjustment for checkpoint or merge selection.Later candidates add a task-reward term and use a lower learning rate.Source: §E.2 · p. 46
  • During ten distillation steps, token distillation loss falls from 0.0708 to 0.0074, while monitored task reward rises from 0.466 to 0.649 under a distillation-only objective.The reward is monitored but does not enter that objective.Source: §E.2 · p. 46, Table 37 · p. 46

E.3 Full Matched Final-Model Evaluation

The final student is compared with SFT and domain experts across matched tasks and budgets. Grouped merging and budget-matched teacher supervision integrate the three experts into one checkpoint.

E.3 Full Matched Final-Model Evaluation

  • On the full matched problem set, k = 6 accuracy rises from 34.58% under SFT to 39.87% under RL, while LiveCodeBench truncation falls from 21.4–37.7% to 1.3–2.6%.Figure 23 separates response completion from accuracy conditional on completion.Source: §E.3 · p. 47, Figure 23 · p. 47, Table 38 · p. 47

E.4 Model Merging

  • After merging and distillation, the student exceeds its corresponding specialist by 1.65 points in instruction following, 1.37 in mathematics and 2.04 in code.All three paired 95% intervals are above zero; the domain averages weight tasks equally within each budget and average across five budgets.Source: Table 42 · p. 48, §E.4 · p. 48

E.5 Multi-Teacher On-Policy Distillation

  • Multi-teacher on-policy distillation initializes the student from a grouped merge, matches each prompt to its domain expert at the student's loop budget, and uses token-level supervision on student-generated responses.The first ten steps optimize distillation alone; task rewards are logged but excluded from that objective.Source: §E.5 · pp. 48–49, Table 43 · p. 49

F.1 Computational Cost

This section estimates the training-cost overhead of jointly sampling five budgets and measures inference cost across them. The estimated average training cost is approximately 20.2% above the unlooped path per token at 8K.

F.1 Computational Cost

  • Approximately 1.202× the unlooped path per token at 8K is the estimated average training cost of uniformly sampling five budgets.The estimate includes forward and backward attention and active-parameter computation, but excludes communication and activation recomputation.Source: §F.1 · p. 49
  • Latency grows approximately with execution count, while additional memory closely follows KV storage across budgets.Table 44 reports runtime metrics for batch sizes one and eight with 128 output tokens per sequence; measurements use one H800.Source: §F.1 · p. 49, Table 44 · p. 50
  • Inference costs were measured on one 80 GB H800 using Accordion-RL, with separate KV caches for each layer execution and held-out web-span prompts.The setup uses bf16 weights and activations, vLLM 0.24.0, and PyTorch 2.11; five warm-ups precede 20 timed iterations, and reported latencies average two hosts’ medians.Source: §F.1 · p. 49
  • At the 32K workload with batch size eight, peak allocated memory ranges from 24.90 to 30.90 GiB across five budgets on one 80 GB H800.The reported memory includes weights, the KV pool, and CUDA-graph buffers.Source: §F.1 · p. 50

Related papers

Loading 2610.32492v1…