PaperlayerView exact revision on arXiv

Source-linked AI summary

CopyLoop: Fast Looped Transformer Decoding via Context Copying and Shallow Screening

Wenyu Du, Stephen Chung

arXiv:2610.38822v1cs.CLcs.AI

TL;DR

Full-depth decoding of looped Transformers incurs latency that scales with recurrent depth, and existing self-speculative pipelines remain limited by stepwise drafting. CopyLoop combines context-copy proposals with shallow screening and continuous scheduling, achieving 2.98–10.70× speedups on Base models and 2.74–3.52× on Thinking models versus full-depth autoregressive decoding.

  • Problem

    Full-depth looped-Transformer decoding has latency that scales with recurrent depth, while prior self-speculative methods remain limited by sequential drafting overhead.

    Source: §1 · p. 1

  • Method

    CopyLoop uses context-copied multi-token proposals, shallow screening, and continuous recurrent scheduling, with full-depth verification and no additional training.

    Source: §1 · p. 2

  • Results

    CopyLoop achieves 2.98–10.70× speedups over full-depth autoregressive decoding on Base models and 2.74–3.52× on Thinking models, outperforming or closely matching baselines across tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CopyLoop needs only lightweight calibration across architectures and gains more on longer generations with reusable spans, while short responses face pipeline overheads.

    Source: §1 · p. 2

    Evaluation is restricted to one RTX 5090 GPU per worker, and BF16 execution can make predictions and continuations sensitive to batch shapes and schedules.

    Source: §5 · p. 10

Abstract

from arXiv · show

Looped Transformers increase computational capacity through the repeated application of shared blocks, but their recurrent depth makes conventional full-depth decoding expensive. While existing speculative pipelines accelerate generation by processing consecutive token positions at varying depths, they are typically limited by sequential decoding overheads. We introduce CopyLoop, a training-free decoding algorithm that combines low-cost context-copy proposals with shallow screening and continuous recurrent scheduling. Rather than step-by-step generation, CopyLoop proposes multi-token suffixes from repeated context. It then evaluates them early at a shallow recurrent depth to efficiently filter out incorrect branches, processing newer blocks concurrently while earlier blocks are still progressing toward full-depth verification. We evaluate CopyLoop across six model checkpoints from the Ouro, Raven, and Huginn families, covering code generation and multi-step mathematical reasoning. CopyLoop achieves decoding speedups of 2.98x--10.70x over full-depth autoregressive decoding on Base models and 2.74x--3.52x on Thinking models, consistently outperforming or closely matching all competing baselines across evaluation tasks. Furthermore, ablation studies confirm a strong synergy between shallow screening and continuous scheduling, while our analyses show that CopyLoop requires only lightweight model calibration and yields greater speedups on longer generations with reusable contexts.

1 Introduction

Looped Transformers incur decoding latency that scales with recurrent depth, while existing self-speculative pipelines remain limited by stepwise drafting. CopyLoop replaces stepwise drafts with context-copied multi-token proposals, shallow screening, and continuous scheduling.

1 Introduction

  • Prior self-speculative methods draft tokens through intermediate recurrent passes, leaving sequential decoding overhead despite concurrent processing at varying depths.A full-depth prediction verifies each draft, retaining matches and discarding mismatches.Source: §1 · p. 1
  • 2.98–10.70× speedups on Base and 2.74–3.52× on Thinking models versus full-depth autoregressive decoding; CopyLoop outperforms or closely matches baselines across configurations.Evaluations cover code and reasoning workloads across four Base and two Thinking checkpoints.Source: §1 · pp. 2–3
  • CopyLoop combines context-derived multi-token proposals with shallow screening and continuous scheduling, using full-depth verification to output final tokens without additional training.Copied tokens are screened against predecessor predictions; only the longest agreeing prefix continues toward full depth.Source: §1 · pp. 1–2
  • Ablations show strong synergy between shallow screening and continuous scheduling; analyses find lightweight calibration sufficient and greater speedups on longer generations with reusable spans.Short-response tasks see limited acceleration due to pipeline overheads.Source: §1 · pp. 2–3

2 Method

CopyLoop forms candidate blocks from a fresh model-predicted head and a tail copied from repeated context. It screens copied tokens at a shallow depth, schedules surviving positions alongside new blocks, and verifies final tokens at full depth.

Block Construction

  • Each block starts with a fresh model-predicted head and appends up to K tokens copied after the longest matching prior-context suffix; without a match, it falls back to a single-token block.Suffix matches are searched from length four down to one.Source: §2 · p. 3

Shallow Screening

  • At ds < D, copied tokens are checked against predecessor shallow predictions; the longest agreeing prefix survives, the rest is pruned, and the final survivor’s prediction seeds the next block.Pruned suffix states are released immediately.Source: §2 · p. 3

Concurrent Pipeline

  • Surviving positions retain their depth-ds states and advance toward full depth while new blocks enter; active positions advance concurrently using their own KV states.This overlaps shallow screening with deep verification while preserving causal dependencies.Source: §2 · p. 3

Full-Depth Verification

  • At depth D, matching candidate tokens are accepted sequentially; the first mismatch supplies the correct token and discards the invalid suffix and dependent continuations.If the copied tail matches completely, the final position’s prediction adds one accepted token.Source: §2 · pp. 3–4

3 Experiments

Across six looped Transformer checkpoints, CopyLoop accelerates code generation and mathematical reasoning, with results showing gains across Base and Thinking models. Ablations, calibration studies, and workload analyses identify when the method works best and how its settings transfer.

3 Experiments

  • Evaluations cover four Base and two Thinking checkpoints, code and mathematical-reasoning benchmarks, and comparisons with AR, PLD, LoopSpec, and WaveFront.The setup uses matched BF16 precision, prompts, stopping rules, and task-specific output budgets across methods.Source: §3 · p. 4

3.1 Decoding Performance Across Architectures and Tasks

  • CopyLoop delivers 2.98×–10.70× speedups over AR on Base models, outperforming all three baselines in 14 of 16 configurations, and 2.74×–3.52× on Thinking models, exceeding all three baselines across eight panels.Table 1 reports accuracy and speedup across architectures and baselines.Source: Table 1 · p. 5, §3.1 · pp. 5–6

3.2 Ablation Studies on Screening and Scheduling

  • With continuous scheduling, shallow screening boosts throughput 1.87×–4.42× versus no screening; the fully coupled configuration is fastest across all eight panels.Without screening, continuous scheduling yields only 0.75×–1.14×; with screening it yields 1.73×–3.48×.Source: §3.2 · pp. 6–7, Figure 2 · p. 6

3.3 Hyperparameter Sensitivity and Transferability

  • Five of six calibration-derived configurations match evaluation-benchmark optima; the exception, Ouro-1.4B Thinking, has a 3.9% deficit.Calibration selects screening depth and copied-tail limit using 20 datasets.Source: §3.3 · p. 7
  • Calibrated screening depth matches the task-optimal depth in 20 of 24 model-benchmark configurations, with remaining throughput drops under 6.81%; copied-tail limits vary more by task.For Ouro-1.4B, the preferred copied-tail limits across HumanEval+, MBPP+, GSM8K, and MATH-500 are 8, 2, 2, and 6.Source: §3.3 · pp. 7–8
  • Using model-level screening-depth calibration with fixed copied-tail defaults yields mean task-wise throughput deficits of 6.20% versus benchmark-specific best tested pairs.The defaults are K = 6 for Base models and K = 2 for Thinking models.Source: Table 2 · p. 7, §3.3 · p. 8

3.4 Analysis of Reusable Spans and Response Length

  • Saved-history replay finds longer matching spans in code generation than in mathematical reasoning, while adaptive copied-tail limits yield only marginal gains under fixed budgets.The replay analysis uses a copied-tail limit of 8 across HumanEval and GSM8K outputs.Source: §3.4 · p. 8, Figure 3 · p. 9
  • On MBPP+, requests of 64 tokens or fewer experience minor slowdowns, whereas those over 128 tokens achieve substantial speedups, including 1.43× for Ouro-2.6B.Short responses may not amortize context lookup, pipeline admission, and startup overheads; BF16 trajectory shifts can also produce repetitive loops that inflate measured throughput.Source: §3.4 · pp. 8–9

4 Related work

Prior work uses recurrent computation and intermediate predictions to accelerate decoding, while CopyLoop replaces stepwise drafting with context-copy proposals and concurrent screening.

4 Related work

  • Looped Transformers reuse shared parameters across depth, with iterative refinement and attention across recurrent depths offering opportunities for parallel decoding [2] [3] [4] [5] [15] [16].Concurrent LoopSpec and WaveFront process positions at varying depths [6] [7].Source: §4 · p. 9
  • CopyLoop bypasses stepwise drafting through training-free context-copy proposals, shallow screening, and continuous recurrent admission [6] [7] [17].Its factorial analysis examines how screening changes when successors enter before full-depth verification finishes.Source: §4 · p. 9
  • Self-speculative methods use early model states, statistical proposals, or retrieval-based candidates with intermediate review before full verification [18] [20] [21] [22] [23] [24] [25] [26].Examples include LayerSkip [22], Cascade Speculative Drafting [23], and ECHO [26].Source: §4 · p. 9

5 Limitations

The evaluation is bounded by single-GPU hardware and BF16 numerical sensitivity, which can make outputs vary across execution configurations.

5 Limitations

  • Evaluation uses one RTX 5090 GPU per worker, leaving broader hardware validation, including multi-GPU and alternative architectures, for future work.Source: §5 · p. 10
  • BF16 execution makes intermediate predictions sensitive to batch shapes and schedules, so continuations are not strictly deterministic across hardware configurations.Source: §5 · p. 10
  • Different batch shapes or execution schedules can alter intermediate token predictions under BF16 precision.Source: §5 · p. 10

6 Conclusion

CopyLoop combines context-copy proposals, shallow screening, and continuous pipeline scheduling to accelerate looped Transformer decoding. Evaluations report speedups of 2.74×–10.70×, alongside synergy between screening and scheduling and greater gains with reusable context and sufficient response length.

6 Conclusion

  • CopyLoop is a training-free framework coupling low-cost context-copy proposals with early shallow screening and continuous pipeline scheduling.Source: §6 · p. 10
  • Shallow screening is indispensable for viable continuous speculative admission, preventing erroneous tails from flooding the pipeline.Source: §6 · p. 10
  • Lightweight calibration is sufficient when transferring CopyLoop to new model architectures.Source: §6 · p. 10
  • CopyLoop gains more speedup with rich matching context spans and enough response length to amortize startup overheads.Source: §6 · p. 10

7 AI Use

The authors report an agent-discovered finding and reproduce and extend the experiments outside Station. The appendix outlines the single-readout decoder and its optional second-depth extension.

7 AI Use

  • The authors say autonomous agents on Station discovered that copying matching past chunks and shallow screening accelerated looped Transformers across models and benchmarks.Source: §7 · p. 10
  • The authors reproduced and extended the finding outside Station using additional models and datasets, then revised the manuscript for researchers.Source: §7 · p. 10

A Decoding procedure and implementation

A.1 Decoding algorithm

  • The algorithm initializes from the prompt, admits blocks, retains the longest copied prefix agreeing at screening depth, and releases the remaining suffix and descendants.Source: §A.1 · p. 14
  • At full depth, the cached prediction is committed and inconsistent branches are discarded; the prefill token is already accepted and is not emitted again.Source: §A.1 · p. 14
  • An admission limit can trigger removal of a block’s copied tail, leaving only its head for the queue.Source: §A.1 · p. 14

A.2 Branching at a Second Readout Depth

The second-readout branch adds an alternative successor when predictions differ at a deeper readout, but measured throughput gains vary by workload. The study therefore retains single-depth decoding as its main configuration.

A.2 Branching at a Second Readout Depth

  • At d2, a block admits an additional child only when its prediction differs from the same endpoint’s prediction at ds; each child receives a copied tail and shallow screening.Full-depth verification keeps the continuation compatible with the verified prefix, removes incompatible descendants, and creates a correction block if none remains.Source: §A.2 · p. 14

B. Experimental Setup and Evaluation Protocols

  • Timing uses paired repeats on the same GPU, and aggregate decode throughput excludes prefill, first-token generation, startup, and warmup.The response-length analysis instead uses request time, which includes prefill, decoding, and request handling.Source: §A.2 · p. 15, §B.1 · p. 15

A.2 Branching at a Second Readout Depth

  • Second-readout branching improves Base decode throughput by 4.02% on HumanEval+ and 11.66% on GSM8K, but Thinking changes are +1.79% on HumanEval+ and −0.47% on MATH-500.Thinking paired-task bootstrap 95% confidence intervals show no clear throughput improvement; the authors retain the single-depth decoder.Source: §A.2 · p. 15, Table 3 · p. 15

B.2 Models, Main Benchmark Panels, and Scoring

The evaluation specifies model configurations, benchmark panels, scoring rules, and a calibration protocol for selecting CopyLoop settings. Calibration uses fixed examples from separate datasets and transfers selected settings to deployment.

B.2 Models, Main Benchmark Panels, and Scoring

  • The main evaluation covers four Base and two Thinking models, with one screening depth followed by full-depth verification; optional second-readout branching is separate.Table 4 defines recurrent depth and maximum copied-tail length K.Source: §B.2 · p. 16, Table 4 · p. 16
  • Base code tasks use EvalPlus tests, GSM8K uses numerical answer extraction, and MATH-500 uses symbolic checking; Thinking evaluation scores only the final-answer segment.Thinking panels contain 188 problems per model, use greedy generation with an 8,192-token budget, and apply EOS stopping; malformed or empty final answers receive zero.Source: §B.2 · p. 16

B.3 Baseline Implementations and Configuration

  • Baseline comparisons match precision, problems, prompts, stopping rules, and output budgets within each model–task panel.The native implementation uses an SGLang-based recurrent engine, recurrent kernels, and CUDA graphs; the WaveFront port follows its official suppression rule on rejection ticks.Source: §B.3 · pp. 16–17

B.4 Hyperparameter Calibration and Transfer Protocol

  • Calibration tests K ∈ {2, 4, 6, 8} and model-specific screening depths on 20 datasets, using eight fixed examples per dataset; evaluation uses separate eight-example benchmark subsets.The selected configuration maximizes equal-weight geometric-mean throughput over calibration datasets; deployment retains calibrated depth and uses K0 = 6 for Base or K0 = 2 for Thinking.Source: §B.4 · pp. 17–18
  • Deployment deficits compare its configuration with tested optima, while full-grid transfer references each benchmark’s best tested (ds, K) pair.Overall deficits use the geometric mean of candidate/reference throughput ratios, and the reported comparisons are restricted to the tested configurations on the same eight examples per benchmark.Source: §B.4 · p. 19

C Supplementary Experimental Results

Supplementary experiments examine execution-schedule sensitivity, component ablations, and hyperparameter selection. They find numerical variation can alter output trajectories without necessarily changing task accuracy, while calibration and joint optimization usually select the same settings.

C.1.1 BF16 Execution and Output Differences

  • Different execution schedules can produce hidden-state divergences and occasionally shift the final argmax, but the observed localized divergences do not consistently propagate into systemic performance drops.The paper cautions that these diagnostics do not establish a universal cause for every accuracy shift.Source: §C.1.1 · p. 19
  • On a 64-task GSM8K diagnostic, AR, dynamic CopyLoop, and fixed-split CopyLoop all score 59/64 despite changes in individual token trajectories.The diagnostic attributes trajectory differences to execution-induced numerical sensitivity, not a systematic accuracy drop.Source: §C.1.1 · p. 19

C.2 Screening and Scheduling Ablation

  • The screening-and-scheduling ablation crosses screening on/off with continuous/staged scheduling while holding context-copy proposals and state continuation fixed.The supplied results passage identifies Table 9 as the numerical results source but does not state the ablation outcomes.Source: §C.2 · p. 20, Table 9 · p. 21

C.3.1 Calibration, Joint Optimization, and Deployment Configurations

  • Across five of six initial models, calibration and joint optimization select the same configuration; Ouro-1.4B Thinking differs, with a 3.90% deficit for its calibrated pair.Calibration uses 20 datasets, while joint optimization uses the four main evaluation benchmarks.Source: §C.3.1 · pp. 20–21, Table 10 · p. 21
  • Full calibrated configurations show overall deficits of 2.58% to 6.75% against per-benchmark optima; these comparisons differ from deployment-default deficits because they use calibrated K.The full-calibrated-pair comparison and deployment-default comparison answer different questions.Source: §C.3.1 · p. 21, Table 11 · p. 22

C.3.2 Hyperparameter Transfer and Sensitivity (ds and K)

Screening-depth settings often transfer well, but the best copied-tail limit varies across benchmarks. Deployment defaults remain close to tested full-grid optima in the reported panels.

C.3.2 Hyperparameter Transfer and Sensitivity (ds and K)

  • The calibrated screening depth is optimal in 20 of 24 initial model–benchmark panels and all eight panels for the two additional Raven models.The four initial-panel exceptions have throughput deficits of 0.69% to 6.81%.Source: §C.3.2 · p. 22, Table 12 · p. 22
  • The best K varies across benchmarks; Ouro-1.4B prefers K = 8, 2, 2, and 6 across its four benchmarks.Table 13 reports decode TPS at each model’s adopted screening depth and marks the best K per panel.Source: §C.3.2 · p. 22, Table 13 · p. 22
  • Against per-benchmark full-grid optima, the 24 initial panels have a mean deficit of 6.20% and median of 4.60%, while the two additional Raven models have overall deficits of 2.18% and 3.86%.These comparisons allow both screening depth and K to change.Source: Table 13 · p. 22, §C.3.2 · p. 23

C.3.3 Diagnostic Case Study: Output-Trajectory Sensitivity

The diagnostic shows that output trajectories can substantially shift aggregate throughput comparisons. Saved-history replay and response-length grouping provide additional evidence about copied-span survival and speedups, with limits on what these analyses establish.

C.3.3 Diagnostic Case Study: Output-Trajectory Sensitivity

  • At ds = 8, increasing K from 6 to 8 raised Huginn MATH-500 throughput from 78.68 to 108.35 TPS as decoded tokens grew from 4,187 to 5,850 while decode time stayed nearly unchanged.The reported decode times were 53.22 and 53.99 seconds, respectively.Source: §C.3.3 · p. 23
  • Two repetitive K = 8 outputs account for 69.98% of decoded tokens and 40.40% of decode time; excluding them reverses the throughput ranking, though reported benchmarks retain all eight examples.Selected outputs include prompts that reach the output cap with lengthy repetitive passages.Source: §C.3.3 · pp. 23–24

C.4.1 Saved-History Replay

  • Saved-history replay measures copied-run survival rather than online acceptance rates or runtime savings, and pooled positions give longer responses greater weight.The analysis uses context lookup on saved histories with a tail limit of eight tokens.Source: §C.4.1 · p. 24

C.4.2 Response-Length Groups

  • Longer-response groups show greater speedups for both models, consistent with better amortization of fixed overheads, but do not establish a prompt-time routing rule or universal length threshold.Groups use realized LoopSpec output lengths and retain paired tasks for both methods.Source: §C.4.2 · p. 24

Related papers

Loading 2610.38822v1…