PaperlayerView exact revision on arXiv

Source-linked AI summary

Target Regeneration Shrinks the Gains of Draft-Training Objectives in Speculative Decoding

Jungseob Lee, Hogun Seong, Euijun Choi, Sugyeong Eo

arXiv:2610.29013v1cs.CLcs.LG

TL;DR

Draft-training objectives are often credited with gains over cross-entropy on external responses, although many pipelines replace those responses with target-generated ones. The paper tests objectives across external and target-regenerated data, finding that regeneration shrinks their gains and recommending regenerated baselines and comparisons across training lengths.

  • Problem

    It is unclear how much draft-training objectives’ gains over cross-entropy on external responses survive when pipelines replace those responses with target-generated ones.

    Source: §1 · pp. 1–2

  • Method

    The study compares cross-entropy, token-level distillation, and survival-gated distillation across external and target-regenerated responses, with controls, training-length comparisons, and vision and text replications.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    Greedy regeneration shrinks objective gains: distillation retains a small gain and the gate little or none, with the pattern also appearing across a second vision corpus and a text-only replication.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    For objectives intended for regenerated-response pipelines, the authors recommend measuring gains against cross-entropy on target-regenerated responses and comparing external-response gains across training lengths.

    Source: §1 · p. 2, §7 · p. 8

    The study uses fixed subsets of at most 26,196 examples and does not extend to the full corpora; several matched-length treatments and controls cover ALLaVA alone.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Speculative decoding accelerates large language models by verifying drafted tokens in parallel. New draft-training objectives are usually credited with the acceptance they add over cross-entropy on externally written responses, yet many pipelines let the target rewrite them. We ask how much of that gain survives. With a vision-language target and a block drafter, we train three objectives on four versions of one corpus: cross-entropy on corpus tokens, token-level distillation from the target, and distillation gated to positions that would survive verification. Distillation and the gate add large gains on external responses, but after greedy regeneration, distillation adds a smaller gain and the gate little or none. The shrinkage persists across data orders, position weights, verifiers, decoding temperatures and, for distillation, initializations and learning rates. Most of it remains at matched supervised length and at each objective's best measured training length. A second vision corpus shows the shrinkage, as does a text-only replication, for distillation also with an autoregressive drafter. Objective gains should therefore be measured against a target-regenerated baseline and across training lengths. Our code is available at https://github.com/js-lee-AI/TargetRegeneration.

1 Introduction

The paper asks whether draft-training objectives retain their gains when external responses are replaced with target-regenerated responses. It compares objectives across response sources and training lengths, finding that regeneration shrinks their apparent gains and motivating regenerated baselines for reporting.

1 Introduction

  • Target regeneration makes corpus tokens match the target’s greedy predictions more often, so cross-entropy already trains toward those predictions and distillation can add only probability assigned to other tokens at matching positions.The authors therefore expect some objective gains to disappear after regeneration.Source: §1 · p. 2
  • The study compares cross-entropy, token-level distillation, and survival-gated distillation on external and target-regenerated responses, with controls, a horizon study, and a text-only replication.The vision-language study uses a block drafter; the text replication also adds an autoregressive drafter.Source: §1 · p. 2
  • After greedy regeneration, distillation retains only a small gain over cross-entropy and the survival gate little or none, despite both adding large gains on external responses.The shrinkage remains at matched supervised length and at each objective’s best measured training length.Source: §1 · p. 2
  • A second vision corpus and a text-only replication also show shrinkage, including for distillation with an autoregressive drafter.The text replication shows shrinkage for distillation and the gate under greedy and sampled decoding; the autoregressive-drafter result is for distillation under greedy decoding.Source: §1 · p. 2
  • The authors recommend measuring objectives intended for regenerated-response pipelines against cross-entropy on regenerated responses and comparing external-response gains across training lengths.A label control supports their account: on external responses, cross-entropy on target-greedy tokens already yields most of the distillation gain.Source: §1 · p. 2

2 Study Design

The study varies the drafter’s training objective, the source and treatment of responses, and the acceptance measure. These choices let the authors compare objective gains across external and regenerated training data.

2.1 Drafter and objectives

  • The block drafter predicts 15 future token slots in one pass from frozen-target features, with sampled anchors and exponentially weighted slot losses normalized by retained weight.The three objectives vary the slot loss and gate: cross-entropy trains on corpus tokens, distillation uses the target distribution, and the gate retains eligible distillation slots.Source: §2.1 · pp. 2–3

2.2 Response treatments

  • Four response treatments preserve images and prompts while using external responses, full greedy regeneration, truncated regeneration, or full regenerated inputs with loss limited to external-response length.Truncation closes the turn, whereas the loss-window treatment continues the response without supervision; the two differ in available anchors near the cut and batch padding, so neither isolates length alone.Source: §2.2 · p. 3

2.3 Measurement

  • The study measures mean accepted tokens per verification round and compares objective increments across response treatments with a ratio of ratios.A negative ratio of ratios means distillation adds less after regeneration; for the gate, the comparison uses gated distillation over distillation.Source: §2.3 · p. 3

3 Experimental Setup

The experiments evaluate vision-language and text-only targets with distinct corpora, drafters, and verification setups. The vision factorial uses multiple evaluation domains, while controls and a second drafter family broaden the tested settings.

3 Experimental Setup

  • The vision study trains on 26,196-example subsets of ALLaVA and ShareGPT4V, continuing a released DFlash drafter except in one control.The factorial runs for three epochs with γ = 7 and the same learning rate across objectives and response versions; each cell is trained twice.Source: §3 · p. 3
  • Vision acceptance is evaluated on 99 prompts in each of three domains using greedy target decoding and a tree verifier that checks up to 63 drafted tokens per round.Run-level mean accepted tokens is the geometric mean of the three domain values, with 95% percentile intervals from prompt resampling.Source: §3 · pp. 3–4
  • Eight ALLaVA controls vary training choices—including initialization, position weighting, data order, learning rate, anchoring, and distillation support—or evaluation verifier and target sampling.The controls compare external and fully regenerated responses; two also test matched-length versions.Source: §3 · p. 4
  • The text replication trains on 24,552 UltraChat prompts and evaluates a DFlash drafter on seven chat, math, and code tasks at temperatures 0 and 1.A separate EAGLE-3 drafter is trained with cross-entropy and distillation and evaluated at temperature 0.Source: §3 · p. 4

4 Results

Across two vision corpora, distillation and the survival gate add large gains on external responses, but greedy target regeneration sharply reduces those gains. The shrinkage persists under controls, matched supervised length, best measured training schedules, and text-only replication.

4.1 Regeneration shrinks objective gains

  • On ALLaVA, regeneration reduces KD’s gain from 18.0% to 3.7% and the gate’s from 6.1% to no measurable gain; ShareGPT4V shows the same pattern.The ratios of ratios are −9.4% for KD and −5.2% for the gate on ALLaVA, and −12.2% and −4.8% on ShareGPT4V.Source: §4.1 · p. 4

4.2 Shrinkage persists at matched length

  • At matched supervised length, KD adds 6.9% and 6.2% over CE after regeneration, less than half its external increment; the gate’s gain depends on the length-control treatment.Truncation gives the gate a 2.2% gain, while the loss window gives 0.2%; masking turn-closing tokens removes most of that difference.Source: §4.2 · p. 5

4.3 Training length moves objective gains

  • At each objective’s best measured schedule, ALLaVA KD adds 8.4% over CE on external responses and 2.4% on regenerated responses, while ShareGPT4V gains are 10.0% and 0.7%.The corresponding ratios of ratios are −5.6% and −8.4%; on ALLaVA, the KD ratio remains negative across the measured training lengths.Source: §4.3 · p. 6, Table 3 · p. 6
  • Training length changes the apparent KD gain: on regenerated ALLaVA responses it rises from 0.9% after one epoch to 3.7% after three, then reaches 9.6% after six.At each objective’s best measured schedule, the ALLaVA KD gain remains smaller after regeneration than on external responses.Source: §4.3 · pp. 5–6, Figure 4 · p. 6

4.4 A text-only replication

  • In text-only tasks, KD’s gain is smaller after regeneration under greedy and sampled decoding, and the same shrinkage appears with an autoregressive drafter under greedy decoding.For the block drafter under greedy decoding, KD adds 7.3% over CE externally and 1.0% after regeneration; the autoregressive drafter’s corresponding gains are 13.7% and 2.2%.Source: §4.4 · p. 6, Figure 5 · p. 7

5 Why the Increments Shrink

Target-written responses already align corpus tokens with the target’s greedy predictions, leaving less room for objectives that address mismatches. The paper separates which part of distillation’s gain survives and shows that even partial regeneration reduces gains.

Why increments shrink

  • Corpus-token agreement with the target’s greedy prediction rises from 58.0% to 95.2% on ALLaVA and from 64.7% to 94.9% on ShareGPT4V after full regeneration.With few mismatching continuations left, CE already trains toward the target’s greedy prediction and the gate has fewer continuations to remove.Source: §5 · p. 7
  • Replacing just one quarter of external ALLaVA responses with greedy regenerations accounts for about two thirds of KD’s decline from external to fully regenerated data.Both objective gains decline as the regenerated share grows, while agreement rises.Source: §5 · p. 7
  • On external ALLaVA responses, CE on target greedy tokens adds 9.3% over CE on corpus tokens, and KD adds a further 4.7%; after regeneration, the same control adds no measurable gain over CE.KD then adds 3.9% beyond the control, close to its 3.7% gain over CE.Source: §5 · p. 7
  • After regeneration, most of KD’s remaining gain lies in how the drafter ranks other tree candidates rather than in correcting its first choice.On external responses, KD mostly corrects the drafter’s first choice.Source: §5 · p. 7
  • Fully regenerated CE exceeds every tested external-response objective setting, including the best soft survival weighting, by 8.5%.Every regenerated cell ends at least 22.9% above the released drafter’s 2.835, while three-epoch external CE ends 1.4% higher.Source: §5 · p. 7

6 Related Work

Related work covers alternative drafter architectures and objectives, target-generated training responses, and distillation beyond drafting. This paper treats response source as an experimental factor rather than proposing a new relabeling method.

Drafters and objectives

  • Draft-training work includes autoregressive and parallel drafters, token-distribution distillation, and losses that downweight unlikely continuations.PARD-2 weights tokens by cumulative target confidence, while GRIFFIN masks loss after a corpus token outside the drafter’s top-k predictions; this paper’s gate is a hard counterpart of PARD-2 weighting.Source: §6 · pp. 7–8

Who writes the training responses

  • Target-generated responses are established in draft training, while other work compares objectives on fixed corpora; this paper instead compares the same objectives across response sources.The paper distinguishes student-generated data with target feedback as a different training factor.Source: §6 · p. 8

Distillation data beyond drafting

  • Sequence-level distillation trains on teacher-generated sequences, whereas word-level distillation matches token distributions; prior work reports that their effects can depend on student capacity and data construction.In translation, adding word-level distillation to sequence-level distillation changed BLEU by less than one point for every student.Source: §6 · p. 8

7 Discussion

The discussion concludes that objective gains depend on who wrote the training responses: distillation and the survival gate lose much of their external-response gain after greedy regeneration. It recommends evaluating regenerated-response baselines and comparing external-response gains across training lengths.

Discussion

  • Even at matched supervised length or each objective’s best measured training length, regenerated-response distillation gains remain less than half the external gains.The text-only replication shows the shrinkage too, including for distillation with an autoregressive drafter.Source: §7 · p. 8

Which gains survive regeneration

  • The gate and soft survival weighting discount continuations the target would not produce, so greedy regeneration leaves few such continuations; KD retains a small gain from probability mass beyond the greedy token.HASS is cited as a different method that retains gain over EAGLE-2 on target-generated responses.Source: §7 · p. 8

Reporting objective gains

  • For objectives intended for regenerated-response pipelines, measure gains over CE on target-regenerated responses and compare external-response gains across training lengths or best schedules.The paper reports that external KD gains roughly doubled from the shortest to the longest measured schedule.Source: §7 · pp. 8–9

Limitations

The study’s conclusions are bounded by its drafter and target choices, limited corpus and training-length coverage, measurement setting, and modest run replication. Several objective and response-treatment variants remain untested or confounded.

Drafters and targets

  • Only DFlash-style drafters are tested in vision; text also tests EAGLE-3, while other drafter families and targets may respond differently.The gate is not tested with an autoregressive drafter, and most vision runs continue training released drafters; random initialization yields smaller-magnitude distillation shrinkage after three epochs.Source: Limitations · p. 9

Corpora and training lengths

  • Limited compute restricts each corpus to at most 26,196 examples, and the study does not extend training to the full corpora.ShareGPT4V has only external and fully regenerated responses; matched-length versions and controls cover ALLaVA alone, while the gate’s horizon study also covers only ALLaVA.Source: Limitations · p. 9

Measurement

  • MAT depends on the verification decoder, and the study does not measure wall-clock speed or answer quality.Vision acceptance is measured against greedy target tokens computed within batched bf16 verification passes.Source: Limitations · p. 9

Runs and intervals

  • Two runs per cell provide only a coarse account of run variation, and no interval covers variation across corpora.Bootstrap intervals mainly capture prompt variation for trained checkpoints; data-order variation is included only for selected cells.Source: Limitations · p. 9

Objectives and treatments

  • The factorial tests one hard survival gate, divergence, γ value, and learning rate; response treatments also change content, length, turn boundaries, and anchor support together.Soft weighting, γ = 2, and the lower learning rate receive narrower testing, so other divergences and position weights may shrink differently; matched-length controls bound but do not isolate length’s role.Source: Limitations · p. 9

Ethical Considerations

The appendices document the study’s ethics, data and model setup, evaluation, controls, text replication, and artifact licenses. The experiments use fixed subsets and specified public evaluation prompts and decoding procedures.

Ethical Considerations

  • The work uses public datasets and released models within their licenses, involves no human subjects or collected human data, and reports that drafting changes generation speed rather than target behavior or risks.Artifacts are used only for non-commercial research.Source: Ethical Considerations · p. 9, §A.5 · p. 14

A.1 Vision training

  • Vision training uses fixed 26,196-example subsets from ALLaVA and ShareGPT4V, with specified filtering, response versions, and a 32-supervised-token minimum.The five-layer drafter reads target layers 1, 9, 17, 25, and 33; regeneration is greedy, while truncation and loss-window treatments differ in whether the response input is cut or only its loss is masked.Source: §A.1 · p. 12

A.2 Vision evaluation

  • Vision evaluation uses fixed COCO, TextVQA, and DocVQA prompt ranges, with a tree verifier that accepts the target-greedy path as far as its drafted tree allows.MAT is computed from accepted paths and the target’s own token, with final-round clipping potentially making it differ from delivered tokens.Source: §A.2 · pp. 12–13

A.3 Vision controls

  • Vision controls vary initialization, weighting, data order, learning rate, anchoring, distilled-token count, verifier, and target sampling; a separate control masks turn-closing losses after truncation.The chain verifier checks one candidate at each of seven slots; sampled evaluation uses temperature 1 with shared random numbers.Source: §A.3 · p. 13

A.4 Text study

  • The text study trains DFlash on a fixed UltraChat subset and evaluates seven tasks; a separate EAGLE-3 replication trains CE and KD and evaluates at temperature 0.Regenerated text responses average 594 supervised tokens versus 344 for originals, and 56.6% reach the 768-token limit.Source: §A.4 · pp. 13–14

A.5 Artifact licenses

  • The listed models and datasets carry specified public licenses, and the paper states that every artifact is used for non-commercial research.The license list includes Apache 2.0, MIT, CC BY 4.0, and CC BY-NC 4.0 artifacts.Source: §A.5 · p. 14

B Vision Results by Domain and Schedule

Vision results show that the apparent objective gains and their shrinkage vary with response source and training schedule. The reduction in distillation’s advantage persists across corpora and run-resampled intervals.

Vision results by domain

  • With runs resampled, full-regeneration ratios of ratios on ALLaVA are [−10.6, −8.1] for KD and [−6.2, −4.2] for the gate; all Figure 3 ratios remain below zero.On ShareGPT4V, the corresponding intervals are [−13.3, −11.0] for KD and [−6.0, −3.7] for the gate.Source: B Vision Results by Domain and Schedule · p. 14

Length controls

  • The gate’s ratio of ratios between truncation and the loss window is 2.0% [0.7, 3.4] with runs resampled, while KD’s corresponding contrast remains unresolved at 0.7% [−0.6, 2.0].The treatments supervise the same number of response tokens but differ at the cut; the table reports CE / KD values by corpus, response source, and training length.Source: B Vision Results by Domain and Schedule · p. 14, Table 5 · p. 15

Schedule horizon

C Controls of the Vision Results

Vision controls show that distillation shrinkage is not eliminated by longer training, alternative initialization, data order, verifier, or temperature. Some controls alter the magnitude or reveal unresolved differences.

Initialization and training length

  • After three epochs, random-initialized KD has a −5.8% ratio of ratios versus −9.4% from released initialization; after six epochs its ratio is −8.5%, negative in every domain and run.Three-epoch random-initialized cells are 26% to 31% below the corresponding released-drafter cells.Source: C Controls of the Vision Results · p. 15

Distillation controls

  • Distilling 64 rather than eight target tokens changes regenerated KD MAT by −0.3% [−0.9, 0.4], with ratio of ratios −9.9% versus −9.4%; an L2 anchor leaves 81% [71, 93] of KD shrinkage.The L2 anchor holds relative weight drift to 0.14 to 0.16, about half that of factorial runs.Source: C Controls of the Vision Results · p. 16

Data orders and verifier

  • Across three data orders, the gate changes MAT after regeneration by −0.4%, 0.2%, and 0.8%, each with an interval including zero; ratio-of-ratios standard deviations are 0.5 points for KD and 0.7 for the gate.Every chain-verifier MAT is lower than its corresponding tree-verifier MAT because each round checks one candidate at seven slots.Source: C Controls of the Vision Results · p. 16, Table 9 · p. 16

Sampled responses

  • Training on three sampled responses per prompt reduces KD’s gain over CE from 4.5% to 1.1% [0.6, 1.6], a ratio of ratios of −3.2% [−4.0, −2.5].Three samples remove 0.75 [0.63, 0.86] of KD’s gain; CE and KD increases are 5.9% [5.1, 6.6] and 2.4% [1.8, 3.1], respectively.Source: C Controls of the Vision Results · p. 16

D Additional Analyses

Additional analyses show how objective gains change across mixtures of external and regenerated responses, corpus versions, and external-training settings. Corpus diagnostics show that the remaining disagreements after regeneration concentrate in uncertain slots, while objective-setting comparisons identify a strong soft-weighted result on external responses.

D Additional Analyses

  • Both KD-over-CE and gate-over-KD gains fall in every domain at the first mixture step, though at least one rises in some domain at later steps.The mixtures retain the 26,101 external examples and their order; the fully regenerated point uses its own 26,193 examples and order.Source: D Additional Analyses · p. 16, Table 10 · p. 16
  • KD’s chain retains 0.78 [0.73, 0.83] of its tree gain on external responses but 0.28 [0.14, 0.43] after full regeneration.The mean top-8-minus-top-1 increment is 1.6 points [1.4, 1.8] externally and 0.3 points [0.2, 0.4] after regeneration.Source: D Additional Analyses · pp. 16–17, Table 11 · p. 17

E Corpus Diagnostics

  • Table 12 reports corpus diagnostics for retained examples, including agreement, positions and weight kept, soft weight, and corpus-token probability.Positions and weight kept are measured for γ = 7 and γ = 2; intervals use 10,000 resamples of examples.Source: E Corpus Diagnostics · p. 17, Table 12 · p. 17
  • After full regeneration, slots with top-1 probability below 0.5 make up 14.5% of slots but 74.1% of disagreements; at least 0.9, disagreement is 0.06%.Truncation inserts a closing token the target never predicts at the cut, accounting for 14.1% of disagreements in the truncated version.Source: E Corpus Diagnostics · p. 17
  • Agreement does not reliably track KD gain: matched-length versions agree within one point of full regeneration yet retain larger gains, while sampled versions have lower agreement but similar KD gains.ShareGPT4V external captions agree with the target more often than ALLaVA external responses, yet KD gains more over CE on ShareGPT4V.Source: E Corpus Diagnostics · p. 17

F External Objective Settings

  • PARD-2 soft survival weighting reaches 3.542 MAT on external ALLaVA, 5.3% above KD with γ = 2 in the main order and 4.6% and 4.4% in two further orders.Under sampled decoding, the four settings keep their greedy-decoding order.Source: F External Objective Settings · p. 17, Table 13 · p. 17

G Text Results

In text-only experiments, distillation gains over cross-entropy shrink on target-regenerated responses, and the survival gate’s relative gain is negative across tasks and temperatures. This pattern holds for both DFlash and EAGLE-3, while greedy-label controls show little additional KD gain over regenerated labels.

G Text Results

  • Every fine-tuned DFlash drafter remains below its released initialization, with losses concentrated on math and code, which the chat corpus barely covers.On MT-Bench and Alpaca, KD’s ratio of ratios is −8.2% under greedy decoding and −8.4% under sampling.Source: G Text Results · p. 18
  • For DFlash, the gate’s ratio of ratios is negative across all seven tasks, both runs, and both temperatures; its regenerated-response gain interval includes zero at either temperature.The intervals are [−4.9, −3.6] under greedy decoding and [−6.0, −2.3] under sampling.Source: G Text Results · p. 18
  • The greedy-label control adds 7.4% [6.9, 7.8] over CE on external responses and 0.3% [0.1, 0.6] on regenerated responses, while KD adds 0.0% [−0.3, 0.3] and 0.7% [0.4, 1.0], respectively, over greedy labels.The label control’s ratio of ratios is −6.5% [−7.0, −6.1] under greedy decoding and −6.3% [−7.6, −5.0] under sampling.Source: G Text Results · p. 18
  • For EAGLE-3, KD adds 13.7% [13.2, 14.3] over CE on external responses and 2.2% [1.8, 2.5] on regenerated responses, a ratio of ratios of −10.2% [−10.7, −9.6].The ratio of ratios is −10.7% and −9.7% in two data orders, and ranges from −12.1% on GSM8K to −6.9% on HumanEval, with every task’s interval below zero.Source: G Text Results · pp. 18–19

Related papers

Loading 2610.29013v1…