PaperlayerView exact revision on arXiv

Source-linked AI summary

ECHO: Efficient Causal Self-Recursion Beyond Auxiliary Multi-Token Prediction

Yuxuan Zhou, Junyuan Shang, Tianyu Wang, Chunrui Zheng, Zichang Liu, Wenbin Chang, Shuohuan Wang, Dianhai Yu

arXiv:2610.24766v1cs.CL

TL;DR

Existing MTP methods direct multi-horizon losses mainly to auxiliary modules, while deeper horizons add modules and compute. ECHO reuses the backbone’s final layer and samples training depth; it reports downstream gains where auxiliary MTP saturates and stronger drafting beyond trained depth, including on open-ended, high-entropy generation.

  • Problem

    Auxiliary MTP gives the backbone multi-horizon learning signals only indirectly, and each added horizon costs another module and compute pass.

    Source: §1 · p. 2

  • Method

    ECHO recursively reuses the backbone’s final layer to predict future horizons and uses Dynamic Depth Sampling to reduce amortized training overhead.

    Source: §1 · p. 2

  • Results

    ECHO improves with added depth where auxiliary MTP saturates, and ECHO-1.5(3) matches ECHO-3’s downstream gain at half the MTP overhead.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    At inference, the pretrained model drafts beyond its trained depth and outperforms post-hoc drafters on open-ended, high-entropy generation.

    Source: §1 · p. 2, §7 · p. 9

    Pretraining was repeated once and finetuning used a single seed; how far the gains extend and how DDS behaves at greater depths remain open questions.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Existing Multi-token prediction (MTP) methods are auxiliary by design: the multi-horizon loss mostly trains modules attached to the backbone, and each added horizon costs another module and another pass. We propose Efficient Causal Self-Recursion (ECHO), which removes both costs. The backbone's own final layer predicts every horizon, so the supervision lands on the parameters that generate the next token, and Dynamic Depth Sampling (DDS) keeps deeper horizons affordable by drawing the training depth with probabilities that decay along depth. Future prediction thus becomes part of the autoregressive objective rather than a task attached to it, and the gains follow on both sides of it. In pretraining, added depth converts into downstream accuracy where auxiliary modules saturate, concentrating on reasoning and code at half the MTP training overhead of training every depth. At inference, the pretrained model is its own drafter: it drafts beyond the depth it was trained for, and keeps its draft-target alignment on the open-ended, high-entropy generation where post-hoc drafters lose theirs. The advantage survives a doubled token budget, a repeated pretraining run, and supervised finetuning.

1 Introduction

Existing MTP methods attach future-token prediction to auxiliary modules, so their losses shape the backbone indirectly and deeper horizons add compute and parameters. ECHO instead reuses the backbone’s final layer across horizons and samples training depth, reporting downstream gains and stronger drafting without post-hoc drafter finetuning.

1 Introduction

  • Auxiliary MTP losses mainly train dedicated prediction modules, while added horizons require another module and compute pass, limiting large-scale causal MTP commonly to one draft token.NTP itself supervises only the immediate next-token horizon.Source: §1 · p. 2
  • ECHO reuses the backbone’s final layer to predict future horizons and samples prediction depth with decaying probabilities, replacing horizon-specific auxiliary modules.This makes multi-horizon supervision act directly on parameters used for next-token generation while reducing amortized training overhead.Source: §1 · p. 2
  • ECHO’s downstream performance continues improving at depth 3 where DeepSeek’s MTP does not, with substantial gains on reasoning tasks.Final-layer sharing sends multi-horizon supervision to the same parameters responsible for next-token generation.Source: §1 · p. 2
  • After supervised finetuning, ECHO’s advantage grows further, and it outperforms post-hoc drafters especially on open-ended generation.It drafts beyond its trained prediction depth while maintaining high acceptance rates, without post-hoc drafter finetuning.Source: §1 · pp. 2–3

2 Related Work

The related work distinguishes auxiliary MTP, post-hoc drafters, and looped transformers by how they produce future tokens and train their parameters. ECHO combines causal recursion with joint pretraining, directly supervising the backbone while addressing costs and alignment limits of auxiliary approaches.

2.1 Multi-Token Prediction

  • Parallel MTP predicts future positions independently, whereas causal MTP preserves dependencies among future steps using dedicated prediction modules.DeepSeek-V3 feeds modules prior hidden states and shifted token embeddings; GLM-5 shares prediction parameters across depths.Source: §2.1 · p. 3

2.2 Post-hoc Drafter

  • Post-hoc drafters are finetuned on target-generated responses, but their limited training data constrains alignment and scaling their multi-step or multi-layer training is compute-heavy.ECHO instead jointly pretrains with the target, aligning drafting and supervising the backbone.Source: §2.2 · pp. 3–4

2.3 Looped Transformers

  • Unlike looped transformers, which recurse over the same token position under next-token prediction, ECHO recurses across future-token horizons.The passage contrasts ECHO’s gains on mathematics, reasoning, and code with maintained performance on multiple choice and closed-book QA.Source: §2.3 · p. 4

3 ECHO: Efficient Causal Self-Recursion

ECHO reuses the backbone’s final layer for causal future-token prediction and uses Dynamic Depth Sampling (DDS) to make deeper supervision affordable. The reported comparisons show downstream gains from shared supervision and a matched benchmark average at half the training overhead with DDS.

3.1 Direct Multi-Horizon Supervision via Final-Layer Sharing

  • At matched prediction depth, ECHO leads DeepSeek’s MTP at depth 1, most clearly on math, reasoning, and code, and reaches +2.63 on reasoning and +2.23 on code at depth 3.Final-layer sharing concentrates supervision from all horizons on parameters that generate the next token.Source: §3.1 · p. 5, Figure 3 · p. 5
  • From depth 1 to depth 3, deeper supervision improves ECHO by 0.63 points averaged over fourteen benchmarks, compared with 0.16 for DeepSeek’s MTP.The passage attributes the difference to ECHO’s backbone receiving each new horizon’s loss directly, while auxiliary MTP mainly trains its newly added module.Source: §3.1 · p. 5, Figure 3 · p. 5

3.2 Affordable Long-Horizon Supervision via Dynamic Depth Sampling (DDS)

  • DDS samples a prediction depth per optimizer step and unrolls the recursion only through that depth, making long-horizon supervision affordable without sacrificing downstream benefit or drafting quality.With π = (0.6, 0.3, 0.1) and K=3, expected depth is 1.5, half the overhead of training all three depths.Source: §3.2 · p. 5
  • 36.34 vs. 36.36: ECHO-1.5(3) matches ECHO-3 on the benchmark average at half the MTP training overhead.DDS still samples depth 3, but does not train every depth at every step; ECHO-1.5(3) also leads MTP-3 on math, reasoning, and code at twice its budget.Source: Figure 3 · p. 5, §3.2 · p. 6

4 Pushing the Frontier of Speculative Decoding under Realistic Serving

ECHO uses Dynamic Depth Sampling (DDS) to concentrate training on shallow depths while shared-layer recursion carries prediction beyond the trained horizon. Its causal, jointly pretrained drafter retains an advantage on open-ended, high-entropy generation, where post-hoc drafters lose alignment.

4.1 Training Shallow, Drafting Deep

  • DDS trains depth k less often as depth increases, matching the fact that rejection at depth k truncates that draft and all later drafts.Unlike reweighting losses while training every depth each step, DDS encodes the decay in how often each depth is trained.Source: §4.1 · p. 6
  • Beyond trained depth K > 3, ECHO keeps gaining at a nearly constant rate, while MTP-3 gains shrink and saturate.ECHO-1.5(3) retains ECHO-3's gain at half the MTP training overhead.Source: §4.1 · p. 6

4.2 Staying Aligned under realistic setting

  • ECHO outperforms post-hoc drafters on realistic, open-ended, high-entropy tasks, with its largest gap on SWE-bench.The authors attribute this to causal drafting keeping draft tokens consistent and joint pretraining aligning the draft distribution with the target.Source: §4.2 · p. 7
  • On high-entropy tasks, causal drafters outperform parallel drafters because they extend drafts step by step as uncertainty rises.On simple code and mathematics tasks, long predictable output stretches can make block-parallel drafting worthwhile; realistic software engineering tasks such as SWE-bench remain high-entropy and favor causal drafting.Source: §4.2 · p. 7
  • ECHO's joint pretraining exposes its drafter to the target's full pretraining corpus, while post-hoc drafters are fit to a frozen target on small finetuning sets.Warm-starting EAGLE-3 improves accepted length over cold-starting, but still falls short of ECHO on open-ended tasks, especially SWE-bench.Source: §4.2 · p. 7

5 The Advantage Holds across Data, Depth and Runs

The section tests whether ECHO's advantage persists with more pretraining data, deeper sampling, and repeated runs. It reports a stable lead under doubled token budget and across runs, with deeper sampling yielding the largest post-SFT lead over DeepSeek's MTP-Instruct.

5.1 Scaling with Data

  • +0.52 at step 78k and +0.50 at step 156k are ECHO-1's leads over DeepSeek's MTP-1; on reasoning & code, the leads are +0.88 and +0.89.The token budget doubles from step 78k to step 156k.Source: §5.1 · p. 8

5.2 Scaling with Depth

  • +2.10 is ECHO-2(7)'s lead over DeepSeek's MTP-Instruct after SFT, the largest among the tested configurations.ECHO-2(7) trails ECHO-1.5(3) in pretraining but overtakes it after SFT by 0.49; it uses 4/3 the MTP training overhead of ECHO-1.5(3).Source: §5.2 · p. 8

5.3 Stability across Runs

  • +0.52 and +0.78 are ECHO's overall leads over DeepSeek's MTP-1 across two pretraining runs, while its reasoning & code lead shifts by only 0.11.Greedy decoding eliminates evaluation noise, leaving training as the remaining noise source examined.Source: §5.3 · p. 8

6 Experiment Setup

Experiments compare ECHO configurations and baselines on a shared 10B-A1B Latent MoE backbone and training recipe. They evaluate few-shot benchmarks and speculative decoding with specified sampling, finetuning, and serving settings.

6 Experiment Setup

  • All configurations use a shared 10B-A1B Latent MoE backbone with 24 layers and the same training recipe to isolate MTP design.The backbone has multi-head latent attention and 256 routed experts, with eight active per token; pretraining uses 600B tokens at sequence length 4,096 on 24 nodes of eight H800 GPUs.Source: §6 · p. 8
  • ECHO-1.5(3) samples depths 1–3 with expected depth 1.5, while ECHO-2(7) samples depths 1–7 with expected depth 2.Every configuration receives the same 5,000-step SFT recipe over one pass of an in-house chat corpus, keeping its pretrained MTP depths.Source: §6 · p. 8
  • Fourteen few-shot benchmarks are decoded greedily and grouped into mathematics, reasoning, multiple choice, code, and closed-book question answering.CoT prompting is used on MATH, GSM8K, and BBH.Source: §6 · p. 8
  • Speculative decoding uses temperature T = 1 for both target and draft, with drafters budget-matched at seven proposed positions plus one bonus token per verification step.Decode-only throughput and mean accepted length are reported on ten tasks split into predictable and open-ended groups by target predictive entropy; post-hoc drafter training and serving use specified H800 GPU setups.Source: §6 · p. 8

7 Conclusion

ECHO makes multi-token prediction part of the autoregressive objective by reusing the backbone's final layer across horizons, while DDS makes deeper supervision more affordable. The authors report gains in downstream performance and speculative drafting that persist across the tested conditions.

7 Conclusion

  • Reusing the backbone's final layer at every horizon directs multi-horizon supervision to the parameters that generate the next token, while DDS trains deeper horizons less often.The paper reports that ECHO improves with added depth, unlike auxiliary MTP such as DeepSeek.Source: §7 · p. 9
  • ECHO-1.5(3) matches ECHO-3's downstream gain at half the MTP training overhead.The conclusion characterizes ECHO as integrating multi-token prediction into the autoregressive objective rather than attaching it as an auxiliary task.Source: §7 · p. 9
  • At inference, ECHO drafts beyond its trained depth and outperforms post-hoc drafters on open-ended, high-entropy generation.The reported advantage holds under a doubled token budget, a repeated pretraining run, and supervised finetuning.Source: §7 · p. 9

Limitations

The authors note limits on stability evidence and open questions about depth scaling. The pretraining setup compares shared-layer ECHO with independent-module MTP and samples prediction depth to manage cost.

Limitations

  • Compute constraints limited pretraining to one repeat and supervised finetuning to a single seed; greater-depth gains and DDS behavior remain open questions.The authors also leave unexplained why deeper sampling pays off only after finetuning.Source: Limitations · p. 9

A.1 Pretraining

  • The models train on 600B tokens with context length 4,096 and global batch size 1,920, using 24 nodes of eight H800 GPUs.The recipe uses bfloat16, Muon with selected AdamW parameters, and an MTP loss scaling factor of 0.3.Source: §A.1 · p. 14
  • ECHO reuses the backbone’s last layer at every MTP depth, while DeepSeek’s MTP uses an independent module per depth.ECHO-1.5(3) samples depths one to three with expected depth 1.5; ECHO-2(7) samples depths one through seven with expected depth 2.Source: §A.1 · p. 14
  • DDS samples one depth per optimizer step, trains all depths up to it, and keeps per-step compute constant by sharing the sampled depth across micro-batches.For depths 1–3, the sampling probabilities are 0.6, 0.3, and 0.1, giving an expected depth of 1.5.Source: §A.1 · p. 14
  • All configurations use the same 5,000-step, one-pass chat finetuning recipe and retain their pretrained MTP depths.This makes the finetuned comparison isolate the same design difference as the pretrained comparison.Source: §A.1 · p. 14

A.2 Benchmark Evaluation

The evaluation specifies benchmark coverage, prompting, scoring, and serving protocols. It also describes a depth-dependent accepted-length analysis and the assumptions behind its training-loss bound.

A.2 Benchmark Evaluation

  • Fourteen few-shot benchmarks cover mathematics, reasoning, multiple choice, code, and closed-book QA, with unweighted averages within groups and across benchmarks.GPQA, SimpleQA, and HMMT are excluded because configurations score near their floor.Source: §A.2 · pp. 14–15
  • Greedy decoding is used throughout; chain-of-thought prompting is applied to MATH, GSM8K, and BBH, while benchmark scores use task-specific rules and macro-averaging.Coding tasks use pass@1 by execution; several QA tasks use an LLM judge, and WinoGrande and PIQA use length-normalized log-likelihood.Source: §A.2 · p. 15

A.3 Speculative Decoding under Serving

  • Serving evaluation splits ten tasks into predictable and open-ended groups by target predictive entropy, excluding tasks with too few verification steps for a stable decode-only ratio.Throughput excludes prefill, and accepted length is decoded tokens per verification step.Source: §A.3 · p. 15
  • The depth sweep reports accepted length at serving depth K as one plus the sum of probabilities that each successive draft prefix is accepted.The expression is exact because the draft loop reaches the next depth only after the current depth is accepted; runs use K=7 and greedy decoding.Source: §A.3 · pp. 15–16

B Decayed Depth Supervision and Accepted Length

  • The bound analysis assumes target-generated training data, with target and data distributions equal, and prefix averages following the training distribution at inference.It assumes acceptance decays with depth; the theorem concerns how tightly training loss bounds lost accepted length, while comparable accepted length at lower DDS cost is verified empirically.Source: B Decayed Depth Supervision and… · p. 16
  • For distributions not generated by the target, the analysis adds a term based on KL divergence between data and target distributions to the bound.The added term depends on neither the drafter nor the sampling schedule.Source: B Decayed Depth Supervision and… · p. 16

C Full Benchmark Results

Across a repeated pretraining run and a doubled token budget, ECHO’s overall lead over DeepSeek’s MTP persists. The reported benchmark table has a reproducibility boundary because five rows use an internal platform with unpublished scoring rules.

C Full Benchmark Results

  • Five of fourteen benchmark rows use internal-platform scores whose answer-extraction and judging rules are unpublished, preventing exact external reproduction.The other nine rows are scored by the authors under the stated evaluation protocol.Source: C Full Benchmark Results · pp. 16–17

D Stability across Pretraining Runs

  • 0.52 points in the baseline run and 0.78 in the repeat: ECHO MTP-1 leads DeepSeek’s MTP-1 across the fourteen benchmarks.Reasoning gains are +0.57 then +1.33, while code gains are +1.35 then +1.69; mathematics moves the other way in the repeat.Source: D Stability across Pretraining Runs · p. 17, Table 2 · p. 18
  • +0.52 at step 78k and +0.50 at step 156k: ECHO’s overall lead holds when the token budget is doubled.The category ordering is reproduced, and every bucket is within 0.2 points across the two budgets.Source: D Stability across Pretraining Runs · p. 17, Table 2 · p. 18
  • Table 2 compares one recipe under two perturbations against the Table 1 baseline; the repeat is an independent same-recipe run at the same step, while twice the budget continues the baseline to step 156k.Source: Table 2 · p. 18

E Next-Token Loss Does Not Separate the Two Objectives

The two objectives reach similar next-token losses, which do not rank their downstream performance. ECHO retains its lead after supervised finetuning, while the loss comparison shows where deeper sampling changes the training terms.

E Next-Token Loss Does Not Separate the Two Objectives

  • Backbone next-token loss differs by 0.004 on average (0.004–0.005 per 10k-step window), while the MTP loss gap reaches 0.011 at step 78k.ECHO-1 is marginally above MTP-1 throughout; the next-token losses are indistinguishable on the same token budget.Source: Figure 5 · p. 18, E Next-Token Loss Does Not Separate… · p. 18
  • Matching next-token cross-entropy is expected because both objectives optimize it; multi-horizon supervision instead constrains the shared representation to support future positions.The passage describes this constraint as regularization and points to downstream benchmark gaps and draft quality as places the objectives separate.Source: E Next-Token Loss Does Not Separate… · pp. 18–19
  • Across five configurations, backbone loss spans 0.006; MTP-3 has the lowest backbone loss, 1.676 versus ECHO-3’s 1.682, despite ECHO-3’s 0.97-point benchmark lead.The benchmark comparison is over fourteen tasks.Source: Figure 6 · p. 19, E Next-Token Loss Does Not Separate… · p. 19
  • At step 78k, ECHO-1.5(3)’s depth-2 and depth-3 terms are 0.051 and 0.132 above ECHO-3’s, while its depth-1 term is 1.874 versus 1.902.The fourteen-task average is 36.34 versus 36.36, and accepted length holds beyond the trained depth.Source: E Next-Token Loss Does Not Separate… · p. 19

F After Supervised Finetuning

  • After supervised finetuning, ECHO leads DeepSeek’s MTP baseline by +1.61 and +2.10 points over the fourteen benchmarks, with larger gains on reasoning and code than QA and multiple choice.The benchmarks are scored under the protocol of Table 1.Source: F After Supervised Finetuning · p. 19, Table 3 · p. 20

G Decode Speedup under Serving

H Comparability of Reported Baselines in Inference

Reported inference baselines differ in target-model scale, training data, drafter capacity, and evaluation configuration. The comparisons also include licensing and redistribution boundaries for the experimental artifacts.

H Comparability of Reported Baselines in Inference

  • No paper in the parallel-drafter family evaluates a dense target above 14B; the largest reported target is sparse Qwen3-Coder-30B-A3B, with roughly 3B active parameters.EAGLE-3 evaluations range from Vicuna-13B to LLaMA-3.3-70B, while parallel drafters cluster at smaller targets.Source: H Comparability of Reported Baselines… · p. 20, Table 4 · p. 20
  • DFlash and Domino used EAGLE-3 checkpoints trained with different datasets and fewer samples; block-parallel drafters also carry 2.1–2.6× its parameters.Drafter parameter counts are computed from released configurations.Source: Table 5 · p. 20, H Comparability of Reported Baselines… · p. 20
  • DSpark matches training data but evaluates EAGLE-3 without tree construction, top-k expansion, or candidate reranking, so its acceptance length understates the published method.The evaluation uses chain drafting rather than the original tree drafting.Source: H Comparability of Reported Baselines… · pp. 20–21
  • ECHO-1.5(3) samples depth 2 on 39.5% of steps; its depth-2 loss series has 6,721 points versus 17,000 for ECHO-3, without evidence of sampling bias.At depth 1, the loss difference between steps that do and do not sample depth 2 is +0.0002, against a standard error of 0.0004.Source: Figure 7 · p. 21

I Artifact Licensing

  • The pretraining corpus and in-house SFT dataset are proprietary and restricted from redistribution; the fourteen evaluation benchmarks are used only for research and are not redistributed.Use follows applicable licenses and access conditions.Source: I Artifact Licensing · p. 21, Table 6 · p. 22
Loading 2610.24766v1…