Beneath the Tokens: A Performance Engineering Study of Multi-Token Prediction in GPU-Accelerated LLM Inference

• The study investigates whether greater speculative token progress offsets the additional work that can make wall-clock performance differ from decoding-algorithm expectations.Source: §1 · p. 1
• Two-token MTP is evaluated across plain-text, reasoning-intensive, and tool-calling workloads using a frozen protocol, application measurements, runtime acceptance metrics, and matched profiler captures.Source: §1 · p. 1
• The analysis asks how MTP affects latency and throughput, how acceptance relates to performance, and how execution changes help explain realized acceleration.Source: §1 · p. 2
• The study contributes a controlled comparison of autoregressive and MTP decoding, a theory-guided evaluation of speculative progress, and cross-layer analysis linking application measurements to GPU execution.Source: §1 · p. 2

• The target model verifies draft-model candidate tokens together and determines how many proposed tokens can be accepted, rather than generating each token in a separate target-model call.Source: §2 · p. 2
• At each speculative position, a proposed token is accepted with probability based on the target and draft distributions; proposals are accepted in prefix order, and later proposals are discarded after a rejection.Source: §2 · pp. 2–3
• One speculative step emits between one and γ + 1 tokens; under an independent, identical acceptance assumption, expected output length is expressed using mean acceptance probability α.Source: §2 · p. 3
• The expected token count includes the target-supplied token and terms for accepting progressively longer draft prefixes; the evaluated configuration uses γ = 2.Source: §2 · p. 3
• The analytical speedup is conditional, because draft cost depends on hardware and software implementation and the model’s assumptions do not guarantee a fixed deployment speedup.Source: §2 · p. 3
• The study evaluates two-token MTP with a dedicated drafter and target-model verification to assess whether speculative progress offsets execution cost.Source: §2 · p. 3

3 Experimental Methodology

• The frozen experiment separates clean application benchmarks, which provide quantitative results, from runtime logs and profiler traces used to examine acceptance and GPU behavior.Source: §3 · p. 4

3.1 System Under Test

• The system uses a quantized Gemma 4 target and a dedicated Gemma 4 assistant drafter for two-token proposals, with common serving parameters across autoregressive and MTP conditions.Source: §3.1 · p. 4

3.2 Workloads

• The three workload classes are plain-text generation, reasoning-intensive generation, and tool calling, each with fixed prompts and output-specific validity rules.Source: §3.2 · p. 5
• Normal and MTP runs use the same prompt content and configuration, with paired experiment identifiers linking equivalent runs for analysis.Source: §3.2 · p. 5

• 360 measured requests covered two decoding modes, three workloads, three prompts, and 20 repetitions.Source: §3.3 · p. 5
• Each mode-workload group began with three unmeasured warm-ups, and autoregressive runs were completed and validated before the MTP service started.Source: §3.3 · p. 5
• Requests ran sequentially with protocol-fixed sampling and output settings, and raw streams, outputs, usage, timestamps, validity, and derived summaries were preserved.Source: §3.3 · p. 5

3.4 Measurements and Validity

• A meaningful output event is reasoning content, final-answer content, or a tool-call fragment; metadata-only streaming events do not count.The client records timestamps for output-related events and request boundaries.Source: §3.4 · pp. 5–6
• The measurements include time to first output, end-to-end latency, completion length, output throughput, and durations associated with reasoning, final-answer, or tool-call phases.Phase durations are associated with client-observed stream boundaries and do not attribute every operation exclusively to one semantic phase.Source: §3.4 · p. 6
• All 360 requests passed execution and phase-mapping validity checks, while 349 also passed output validation.The eleven reasoning requests that failed output validation were retained rather than replaced.Source: §3.4 · p. 6
• The analysis compares matched prompts and interprets end-to-end latency alongside completion length and output throughput.Analyses requiring a correct, completed trajectory use only output-valid requests.Source: §3.4 · p. 6

3.5.1 Nsight Systems

• Six Nsight Systems captures compare repeated GPU execution cadence, inter-group gaps, kernel composition, invocation counts, and aggregate kernel duration.Each capture used a temporary vLLM instance, a settling interval, three warm-ups, and one target request.Source: §3.5.1 · p. 6

3.5.2 PyTorch Profiler

• Six matched PyTorch Profiler captures preserve worker, frontend, and summary traces; all twelve profiled target requests passed phase-mapping and output validation.The worker traces provide operational context for repeated model execution and identifiable matrix, attention, and speculative-path operations.Source: §3.5.2 · p. 6
• Table 1 reports workload-level medians across benchmark-valid requests, with 60 requests per row; E2E means end-to-end latency and TPS means output tokens per second.Source: Table 1 · p. 7

3.5.3 Relationship to the Analytical Cost Model

• The study does not estimate theoretical coefficient c directly because the traces lack unambiguous boundaries for measuring isolated draft and target step costs.Where execution groups can be identified consistently, the analysis instead considers their relative duration without treating that ratio as equivalent to c.Source: §3.5.3 · p. 7
• Nsight Compute is used only when Nsight Systems and PyTorch Profiler identify a specific kernel requiring hardware-level characterization.Source: §3.5.3 · p. 7

4.1 Dataset Validity

• All 360 requests passed benchmark and phase-mapping validation; 349 passed output validation, and application timing used all benchmark-valid requests.The eleven output-invalid requests were confined to reasoning-intensive workloads and were retained rather than replaced.Source: §4.1 · p. 7

4.2 RQ1: Application-Level Performance

• Across prompts, two-token MTP raised throughput 1.91×–2.19× and reduced TTFO 10.0–14.2%; tool-calling comparisons were stable, reasoning retained roughly twofold acceleration on output-valid results, and plain-text E2E required length-aware interpretation.Source: §4.2 · p. 8

4.3 RQ2: Acceptance and Theoretical Correspondence

• MTP’s median mean acceptance length was 2.370–2.595 tokens per verification iteration, but realized throughput gains were lower because speculative execution added cost.Acceptance declined from the first draft position to the second in every workload; acceptance indicates useful progress, not a one-for-one prediction of wall-clock speedup.Source: §4.3 · p. 9, Table 3 · p. 9

4.4 RQ3: GPU Execution Analysis

• MTP added auxiliary graph sequences and support operations, yet repeating-graph executions per token fell 56.4–78.1% and summed kernel duration per token fell 52.0–75.5%.MTP’s recurring-group cadence was 16.8–19.7% longer; its dominant BF16 GEMM took approximately 2.784 ms per invocation versus approximately 2.777 ms for AR GEMV, so the reduction came from fewer invocations, not a faster dominant kernel.Source: §4.4 · pp. 11–13, Table 4 · p. 11, Table 5 · p. 12

4.5 RQ4: Runtime Operations and Implementation Costs

• MTP produced 2.403–2.625 completion tokens per steady-state generation context, compared with exactly one completion token per AR context.The reported MTP context counts were 221 for 531 plain-text tokens, 224 for 588 reasoning-intensive tokens, and 124 for 321 tool-calling tokens.Source: §4.5 · p. 15, Table 8 · p. 15
• MTP reduced steady-state generation contexts by 58.5–62.0% and advanced 2.403–2.625 completion tokens per context, while median GPU-context spans were 58.5–61.5% longer.Median CPU-context spans were also 33.3–34.3% longer under MTP.Source: §4.5 · p. 17
• The selected GPU-context span per completion token improved by 1.485–1.650×, below MTP’s 2.403–2.625 completion-token progress per context.These profiled-context ratios are not whole-request latency ratios; the traces were instrumented executions and completion tokens per context are not exact acceptance counts.Source: §4.5 · p. 17
• MTP sample_tokens duration was approximately 2.15–2.29 times the AR duration, while execute_model was approximately one-third longer.All observed proposal-ID ranges were nested within sample_tokens, and proposer.propose ranges were nested within proposal-ID processing.Source: §4.5 · p. 17
• MTP added draft proposal, expanded sampling, candidate selection, gathering, reduction, tensor-manipulation, and state-update work, offsetting part of its token-progress advantage.The proposal path was absent from AR; proposal calls were nested within sample_tokens, so inclusive durations must not be added as independent latency contributions.Source: §4.5 · pp. 16–17

4.6 Selected-Kernel Analysis

• The selected MTP GEMM sustained 575.07 GB/s (95.93% of maximum memory bandwidth), while the AR GEMV sustained 579.04 GB/s (96.59%).Both selected kernels had low L2 hit rates and operated near the A10G’s nominal 600 GB/s DRAM-bandwidth limit; these representative instances came from matched reasoning-intensive captures.Source: §4.6 · p. 17, Table 10 · p. 18
• The MTP GEMM achieved 8.35% occupancy versus 66.33% for the AR GEMV, yet the selected kernels had nearly identical durations and similar bandwidth ceilings.MTP’s workload-level improvement instead came from requiring fewer dominant matrix-kernel and recurring group executions per generated token, despite its more complex, individually costlier execution path.Source: §4.6 · p. 18

5.1 Useful Progress Versus Speculative Cost

• MTP accelerated generation because fewer recurring executions per generated token outweighed the higher cost of each more complex execution.Proposal generation, expanded verification, sampling, and state updates made MTP executions longer; the gain came from amortizing this work, not faster executions or dominant kernels.Source: §5.1 · p. 18

5.2 Analytical Model Versus Implemented Execution

• The implemented-path measurements cannot isolate the analytical model’s draft-cost coefficient c.Measured cadence and context-span ratios include proposal, verification, sampling, scheduling, and device activity, so they are not empirical estimates of c.Source: §5.2 · p. 18

5.3 Practical Implications

• Evaluate accepted progress, execution frequency, and per-execution cost together; acceptance rate or isolated kernel duration alone can mislead about wall-clock speedup.The relevant deployment objective is useful output progress relative to measured end-to-end latency.Source: §5.3 · pp. 18–19

6 Threats to Validity

• The study tested one model pairing, one A10G GPU, two-token speculation, single-request serving, and three prompts per workload.Acceptance telemetry used aggregate windows; profiling covered one matched request per workload and mode, and Nsight Compute examined one selected GEMM and GEMV instance.Source: §6 · p. 19

7 Future Work

• The study fixed speculative depth at γ=2 and did not aim to identify an optimal depth or characterize depth scaling.Its stated objective was to explain the runtime and GPU-execution behavior of a deployed two-token MTP configuration.Source: §7 · p. 19
• Future depth-scaling studies could track acceptance, recurring GPU-execution cadence, proposal-processing overhead, and end-to-end performance across γ values.Such measurements could characterize the trade-off between speculative progress and proposal, verification, and acceptance-processing costs.Source: §7 · p. 19

8 Conclusion

• 1.91× to 2.19× higher output throughput across all prompts accompanied 10.0–14.2% TTFO improvements in the evaluated deployment.MTP’s greater token progress more than compensated for its costlier proposal and verification path, reducing recurring GPU executions per generated token rather than speeding the dominant kernel.Source: §8 · p. 19

A Experimental Protocol

• The protocol was finalized before primary measurement and preserved with a SHA-256 integrity record.It is identified as frozen version 1.2.Source: A Experimental Protocol · p. 20
• Unique identifiers were inserted into user messages so paired normal and MTP runs reused identical prepared payloads.The primary condition was controlled_cold; the warm-cache experiment was deferred.Source: A Experimental Protocol · p. 22
• The primary benchmark used three prompts per workload and targeted 20 valid repetitions per prompt.Runs were settled after health checks, with at least three and at most ten requests per workload and two consecutive requests without a JIT warning required.Source: A Experimental Protocol · p. 22
• Recorded metrics included time to first output, end-to-end latency, phase-associated durations, token counts, and output tokens per second.The listed phases included reasoning-associated and final-or-tool-associated duration.Source: A Experimental Protocol · p. 22
• Benchmark validity rules excluded runs with measurement-time JIT, uncontrolled GPU work, service or stream failures, active profiling, or unmet cache conditions.Phase mapping and output also had explicit invalidation criteria; invalid runs were preserved.Source: A Experimental Protocol · p. 22
• Profiling evidence was used to investigate execution behavior, while the benchmark remained the source of quantitative performance results.Matched captures included Nsight Systems timelines, PyTorch Profiler runtime hierarchies, and selected Nsight Compute kernel comparisons.Source: Figure 1 · p. 23, Figure 2 · p. 24, Figure 3 · p. 24