PaperlayerView exact revision on arXiv

Source-linked AI summary

HyCo: Hybrid Drafting for Cost-Efficient Mobile-Cloud Speculative Decoding

Hanbin Bae, Heejin Choi, Joohyung Lee, Byoung Jin Choi, Sangjun Park, Hoon-Young Cho

arXiv:2610.21436v1cs.CLcs.DCcs.LG

TL;DR

Device–cloud speculative decoding seeks to reduce server computation without making mobile delivery impractically slow. HYCO assigns draft-tree depth to the verifier’s native MTP head on the device and alternatives to one SLM pass. In single-client measurements, it lowers server GPU time per output token against Cloud+MTP on both reported tasks.

  • Problem

    Device–cloud speculative decoding offers a way to reduce server computation, but communication can make an individual request slower than server-only speculation.

    Source: §1 · p. 1

  • Method

    HYCO runs the verifier’s native MTP head on the device to provide draft depth and uses one SLM pass to add alternative branches.

    Source: §6 · p. 8

  • Results

    38.6% and 29.2%: HYCO reduces server GPU time per output token versus Cloud+MTP on Spec-Bench and 32K-token summarization, respectively, in single-client measurements.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Hybrid drafting offers a practical mobile–cloud operating point without requiring the SLM alone to provide both depth and alternatives.

    Source: §6 · p. 8

    HYCO requires access to the verifier’s native head, intermediate hidden states, and KV entries, which may be unavailable from closed model services.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Device-cloud speculative decoding reduces server computation, but growing a deep draft tree with a small language model (SLM) requires repeated passes on a mobile neural processing unit (NPU). We present HyCo, which assigns depth to the server model's native multi-token prediction (MTP) head, executed on the device, and alternative branches to one SLM pass. The server returns the hidden state, input embedding, and key-value entries needed by the head instead of executing its sequential steps. Quantization, a restricted vocabulary, and a 1K-token window fit both drafters to the NPU. An {\epsilon}-threshold rule additionally relaxes exact preservation of the verifier's sampling distribution. In single-client runs with a Galaxy S26 Ultra over LTE/5G, HyCo reduces server GPU time per output token relative to server-side MTP self-speculation by 38.6% on Spec-Bench and 29.2% on 32K-token summarization. With the same acceptance rule, three Spec-Bench runs per system show 9.0 percentage points more saving and 1.9x delivery speed than an SLM-only drafter. HyCo differs from Cloud+MTP by +0.2 judge-score points (paired 95% CI: -0.9 to +1.3); exploratory translation favors SpecEdge-{\epsilon}. Without relaxation, HyCo exact saves 27.7% and 11.9% on the two datasets.

1 Introduction

Device–cloud speculative decoding can reduce server computation, but communication and repeated mobile SLM passes can undermine user speed. HYCO divides drafting between the verifier’s native MTP head for depth and one SLM pass for alternatives, reducing server GPU time in single-client evaluations.

1 Introduction

  • A four-level SpecEdge-style tree raises mean accepted tokens per round, but its four sequential mobile SLM passes leave users slower than the SLM alone.The challenge is to gain useful depth and alternatives without repeatedly running the SLM.Source: §1 · p. 1
  • HYCO assigns depth to the verifier’s native MTP head on the device and alternative branches to one SLM pass, while the server returns the head’s inputs instead of running its sequential steps.The hybrid tree lets depth continue where the MTP chain breaks without requiring extra SLM passes.Source: §1 · pp. 1–2
  • Quantization, a restricted output vocabulary, and a 1K-token window fit both drafters to the NPU, with some cost in acceptance; ϵ-threshold acceptance raises acceptance but changes the output distribution.HYCO evaluates quality alongside server GPU time, mobile draft time, and delivery speed, including exact-verification and same-threshold SLM-only comparisons.Source: §1 · p. 2
  • 38.6% and 29.2%: HYCO reduces server GPU time per output token versus Cloud+MTP on Spec-Bench and 32K-token summarization, respectively.On Spec-Bench, HYCO delivers 35.0 tokens/s, 1.9× the SLM-only baseline’s rate; with exact verification, savings are 27.7% and 11.9%, respectively.Source: §1 · p. 2, Figure 1 · p. 2
  • The contribution is a native-head/SLM division of drafting work and an implementation demonstrating its usefulness under mobile NPU constraints.The server should spend less GPU time than Cloud+MTP while user delivery is at least as fast as the SLM alone.Source: §1 · pp. 1–2

2 Related Work

Prior device–cloud speculative decoding strengthens drafts or skips verification for locally confident tokens. HYCO instead uses the verifier’s native MTP head on the device and raises acceptance within verification using its own threshold rule.

2 Related Work

  • Device–cloud systems strengthen drafts with trees, draft-model training, or parallel drafting; HYCO keeps released drafters apart from quantization and a restricted vocabulary.Training-based approaches are described as orthogonal to HYCO.Source: §2 · p. 2
  • SpecEdge overlaps sequential SLM passes with the round trip through proactive drafting, while HYCO uses one SLM pass for alternatives rather than depth.SLM-based systems draft every proposed token with one sequential pass per tree level [Park et al., 2025; Li et al., 2025a, 2026c].Source: §2 · p. 2
  • HYCO differs from prior device–cloud systems by running the LLM’s native draft head on the mobile device beside one SLM pass; Cloud+MTP runs that head on the server.Other approaches cited either run initial LLM layers on-device or draft from server-returned hidden states.Source: §2 · p. 2
  • Unlike confidence-based verification skipping [Wang et al., 2025], HYCO checks every round and applies ϵ-threshold acceptance within verification, which is also lossy.HYCO’s threshold tests the verifier’s own sampling distribution ˜p.Source: §2 · p. 3
  • Lossy acceptance rules raise acceptance by accepting some tokens the target would not necessarily sample; HYCO thresholds the verifier’s sampling distribution rather than the divergence tests used by BiLD and related methods.Fuzzy speculative decoding and edge–cloud E2-SCI use their own divergence thresholds.Source: §2 · p. 3

3 Method

HYCO constructs a hybrid draft tree on the mobile device for server verification, selecting settings on UC-val by minimizing server cost within an observed judge-score range. Its method combines an on-device MTP chain and SLM alternatives, with quantization, a restricted vocabulary, a short context window, and ϵ-threshold acceptance.

3.1 Hybrid draft tree

  • Moving the MTP head to the device lowers server cost g by 15.2%, from a 26.8 ms to a 21.7 ms median server round, at about 20 ms more per mobile round.The server returns the head’s inputs, totaling 5.6 KB per round plus 6.1 KB per output token at 8 bits.Source: §3.1 · p. 3
  • The MTP Drafter supplies tree depth, while one SLM pass adds alternatives under the root and planted nodes; the verifier checks the trimmed tree in one pass.The MTP head runs sequentially on the device, while the server returns its inputs instead of executing those steps.Source: §3.1 · pp. 3–4
  • The chain’s cumulative acceptance reaches 18.9% at depth 4, and an SLM leaf extends a fully accepted chain to a fifth token in 17% of exact-verification rounds.The selected configuration uses K=4, runner-up siblings at depths 1–2, and a 32-node budget.Source: §3.1 · p. 4

3.2 Quantization

  • Both drafters use integer execution graphs; the SLM’s 16-bit floating-point activations lengthened its 32-row pass from 15.7 to 38–41 ms at unchanged acceptance.The MTP head receives 4-bit quantization-aware training, and both drafters use 16-bit fixed-point activations.Source: §3.2 · p. 4

3.3 Restricted output vocabulary

  • The 86K-token keep set yields g of 6.23 ms, the lowest among four tested sizes, compared with 6.44–7.57 ms.Neither drafter proposes tokens outside the keep set, but the verifier still samples bonus tokens from its full vocabulary, preserving exact verification.Source: §3.3 · p. 4

3.4 1K-token context window

  • A 4,096-token SLM context changes g from 6.82 to 6.77 ms versus 1,024 tokens, while server-side MTP head windows of 512–2,048 tokens yield 7.75–7.89 ms; both differences are within noise.Both drafters use a 1,024-token window.Source: §3.4 · pp. 4–5

3.5 ϵ-threshold acceptance

  • ϵ-threshold acceptance accepts a drafted child when it matches the verifier’s sample or its probability under the verifier’s sampling distribution reaches ϵ; the rule can change selection probabilities.The verifier chooses the deepest accepted path, preferring more verifier-sampled edges and then the highest probability product among equally deep paths.Source: §3.5 · p. 5
  • At ϵ=0.005, applying the threshold at every node lowers g from 5.62 to 5.10 ms at the same judge score of 60.0, and this setting has the highest MAT and lowest g among screened settings.The ϵ=0.02 setting matched that g but scored 57.1 without node logging; on one run’s fixed logged trees, MAT was 4.364 at ϵ=0.005 versus 4.366 if every in-support drafted token were accepted.Source: §3.5 · p. 5

4 Experiments

The experiments compare HYCO with server-side MTP, SLM-only drafting, and other variants across single-client efficiency and answer-quality evaluations. They find lower server GPU time with HYCO, while results also expose trade-offs in delivery speed, quality assessment, and startup delay.

4.3 Evaluation and uncertainty

  • The paired quality intervals condition on recorded generations and one Cloud+MTP run, and do not separate generation variation from judge variation.The evaluation uses all 480 SB questions for efficiency, but quality comparisons cover 398 HYCO–Cloud+MTP questions and 396 complete-case SpecEdge-ϵ questions.Source: §4.3 · p. 6

4.4 Efficiency under matched acceptance rules

  • With exact verification, HYCO saves 27.7% on SB and 11.9% on 32K, compared with 20.6% and 2.9% for SpecEdge-style exact.These comparisons preserve the verifier’s sampling distribution.Source: §4.4 · p. 6
  • HYCO-2P saves 44.7%/31.9% on SB/32K but delivers fewer tokens per second than default HYCO; proactive drafting also lowers delivery speed.The server-head configuration delivers 36.1 tokens/s on SB but saves only 14.6% of server GPU time.Source: §4.4 · p. 6

4.5 Quality and failure cases

  • HYCO differs from Cloud+MTP by +0.2 judge-score points (95% CI: −0.9 to +1.3), while HYCO scores 2.6 points below SpecEdge-ϵ on translation (95% CI: −5.2 to −0.2).The 15 task-wise contrasts are exploratory and unadjusted for multiple comparisons; intervals spanning zero do not establish quality equivalence.Source: §4.5 · pp. 6–7

4.4 Efficiency under matched acceptance rules

  • With the same ϵ rule, HYCO saves 38.6 ± 0.2% of GPU time on SB versus 29.6 ± 0.2% for SpecEdge-ϵ, and 29.2% versus 15.7% on 32K summarization.Delivery speeds are 35.0 ± 0.7 versus 18.9 ± 0.1 tokens/s on SB, and 21.0 versus 12.4 tokens/s on 32K summarization.Source: §4.4 · p. 6, Table 2 · p. 7

4.5 Quality and failure cases

  • Reference-based SB checks report GSM8K accuracy of 95.0% versus 95.0%, RAG contains-gold accuracy of 65.7% versus 67.1%, CNN/DailyMail ROUGE-L of 19.2 versus 19.8, and WMT14 BLEU of 33.9 versus 33.7.These are descriptive single-generation comparisons; RAG measures normalized reference containment, not full answer correctness. On 32K summaries, HYCO’s HELMET F1 difference from Cloud+MTP is +0.4 points (−2.3 to +3.1), while ROUGE-L-Sum differs by −1.0 (−2.1 to +0.1); these measures do not guarantee factual consistency.Source: §4.5 · p. 7

4.6 Mobile costs and head transfer

  • On SB, HYCO delivers 35.0 tokens/s versus 26.4 for the SLM alone, with 12.1 versus 36.8 ms of mobile draft time per output token.Cloud+MTP was not streamed to the phone, so its user delivery speed is not compared with HYCO.Source: §4.6 · p. 7
  • HYCO’s mean time to first token is 846 ms on SB and 4.60 s on 32K inputs, so higher average delivery speed does not imply lower first-token delay.The study reports GPU-busy efficiency, not loaded-service capacity or energy savings.Source: §4.6 · p. 7

5 Discussion

HyCo moves the verifier’s native MTP steps to the device and pairs them with one SLM pass for alternatives, trading server work against mobile work. The discussion compares operating points and notes that ϵ-threshold acceptance retains distributional risk.

5.1 From server accelerator to client drafter

  • HYCO moves the native head’s sequential steps to the device and returns the verifier-state slice it reads, turning MTP self-speculation into a device–cloud protocol.The SLM supplies alternative branches without taking over the chain’s depth.Source: §5.1 · p. 8

5.2 Choosing the mobile–cloud work split

  • Figure 1 illustrates deployment priorities rather than an evaluated online switching policy.The choices include where the native head executes and how many SLM passes accompany it.Source: §5.2 · p. 8
  • HYCO-2P reaches 29.0 tokens/s on SB-480 versus 35.0 for default HYCO; on ∞Bench Sum, its extra pass adds only 2.7 percentage points of server saving and reduces delivery speed by 10%.Prefill, which drafting does not shorten, accounts for about a fifth of g.Source: §5.2 · p. 8
  • Cloud+MTP uses 9.2 ms of mobile draft time per output token versus 14.7 ms for HYCO exact, with 13.1 percentage points less server saving on SB.Time measurements alone do not determine an energy-optimal choice.Source: §5.2 · p. 8

5.3 Lossy acceptance revisited

  • ϵ-threshold acceptance does not preserve the verifier’s relative preferences within the allowed set, and its repeated-generation and long-summary results characterize the trade-off only on evaluated tasks.The same-ϵ comparison controls the acceptance rule but not the proposal distribution; overall score intervals and translation results indicate a task-dependent trade-off.Source: §5.3 · p. 8
  • Threshold-only accepted tokens in the fixed-tree replay came about two thirds from the verifier’s MTP chain, a tenth from runner-up siblings, and a quarter from the SLM.The replay follows fixed logged trajectories and does not measure the quality of newly generated answers; proposed explanations for scores are not guarantees.Source: §5.3 · p. 8

6 Conclusion

HyCo combines the verifier’s native MTP head for draft-tree depth with one SLM pass for alternative branches. In single-client measurements, it reduces server GPU time against Cloud+MTP, while exact verification also retains savings.

6 Conclusion

  • The verifier’s native MTP head supplies depth, while one SLM pass supplies alternative branches; quantization, a restricted vocabulary, and bounded context make the split practical on a mobile NPU.Returning the head’s inputs instead of executing its sequential steps reduces server work, and the hybrid tree avoids repeated SLM passes for depth.Source: §6 · p. 8
  • With ϵ-threshold acceptance, HYCO reduces server GPU time per output token against Cloud+MTP by 38.6% on SB and 29.2% on 32K-token summarization; exact verification saves 27.7% and 11.9%, respectively.These are single-client measurements.Source: §6 · p. 8
  • Applying the split to other model families requires a native head that the mobile device can run; the paper identifies it as a natural next step.Appendix H shows the SLM Drafter working for Muse-Glimmer-30B through a token bridge without retraining.Source: §6 · p. 8

Limitations

The evaluation is limited to single-client deployments and does not establish loaded-service capacity, cost, or energy. Its quality evidence also has task, metric, and sampling limitations, while the approach requires specific model access and device capabilities.

Limitations

  • The deployment uses one mobile device, one commercial cellular network, one cloud GPU type, and one verifier family, with configurations serving one client at a time.The study measures GPU execution time and mobile draft time, not loaded-service capacity, monetary cost, or device and radio energy.Source: Limitations · p. 9
  • The quality assessment does not comprehensively measure factual consistency or output diversity, and translation favors SpecEdge-ϵ in an exploratory comparison.The aggregate covers scored non-QA first-turn answers, not QA or second-turn quality; only the two main relaxed systems have three SB generations, and the other configurations and Cloud+MTP have one.Source: Limitations · p. 9
  • HYCO requires access to the verifier’s native head, intermediate hidden states, and KV entries, and the head must run on the mobile device.These capabilities may be unavailable from closed model services; the SLM also needs a shared tokenizer or token bridge.Source: Limitations · p. 9
  • The UC-derived keep set and 1K window need not suit every language or domain, and the evaluation covers text rather than Gemma’s image or audio inputs.The SLM accesses earlier context only indirectly through the MTP chain.Source: Limitations · p. 9
  • Transmitted activations and reply-size patterns may expose information beyond the text, and the study evaluates no safeguard.Each client holds the MTP head and part of the verifier’s input embeddings.Source: Limitations · p. 9

A The Drafters on the Mobile Device

The appendix describes the mobile drafters’ quantized implementation and calibration, the verifier engine, and the configuration and steps of a decoding round.

A The Drafters on the Mobile Device

  • The SLM graph preserves checkpoint weight integers and scales while rewriting low-bit matrix multiplications as integer operations; the drafters use fixed activation ranges and no floating-point operator fallback.The MTP head is quantization-aware trained using verifier captures, while the SLM and applicable MTP head use UC-tr calibration prompts.Source: A The Drafters on the Mobile Device · p. 13
  • The MTP Drafter’s output layer is restricted to a keep set of 86,089 tokens, while both drafters can accept committed tokens outside that set.The keep set includes special, byte, and Hangul tokens plus frequent UC-tr assistant-turn tokens; hand-off state is 8-bit, with the initial prompt window capped at 1,024 tokens and 6.3 MB.Source: A The Drafters on the Mobile Device · pp. 13–14

B Verifier Engine

  • The verifier uses integer Marlin attention projections and routed fused Marlin MoE kernels, with dense MLP and router operations in 16 bits.Decode rounds use padded attention windows and CUDA graph replay; prefill runs eagerly in chunks.Source: B Verifier Engine · p. 14

C Decoding Round and Configurations

  • The configurations use chain depth K=4, runner-up siblings through D=2, node budget B=32, a 1,024-token window, and an 86,089-token keep set.The listed ϵ is 0.005 except in the exact variant; ϵ=1 reduces the specified acceptance step to exact verification.Source: C Decoding Round and Configurations · p. 14
  • Each HYCO round drafts tokens on the device, builds and sends a hybrid tree, verifies it on the server, then commits the deepest matching path and bonus token.The round advances the position and updates the SLM Drafter’s KV rows for the next pass.Source: C Decoding Round and Configurations · p. 14
  • The mobile MTP Drafter consumes the hand-off embedding, hidden state, and verifier KV rows, then iteratively runs the head to generate tokens and runner-up candidates.Its ring buffer holds W KV columns, and it returns the drafted chain and sibling tokens for hybrid-tree construction.Source: C Decoding Round and Configurations · p. 14

D Validation Protocol and Measurement Conditions

The final configuration was selected on UC-val before the SB and ∞Bench Sum runs, while development-stage sweeps compare configurations only within their respective blocks. Validation reports both server GPU time and mobile timing under specified prefill, context, and test-run conditions.

D Validation Protocol and Measurement Conditions

  • The final 86K keep set was derived from UC-tr and ϵ was chosen on UC-val before final benchmark runs; development-stage sweeps are not repetitions of the final setup.UC-val has 60 prompts, about 100 tokens at the median, and a 512-token output limit; Table 6 selection rows should be compared only within each block because blocks used different sessions and setups.Source: D Validation Protocol and Measurement… · p. 14, Table 6 · p. 16
  • The mobile tree uses the SLM Drafter with an 86K keep set and 1,024-token context, while Table 5 identifies chain depth, sibling depths, node counts, and acceptance settings.PD continues drafting while a verification reply is in flight, and all SLM-based device configurations use the final UC-tr activation calibration.Source: Table 5 · p. 15
  • The hybrid tree plants a K-token chain and runner-up siblings, then uses one SLM forward pass to expand candidates and trims the tree to the B highest-scoring nodes and their ancestors.HYCO-2P repeats the expansion-and-trim stage once more; the returned rows include the root and are padded to B+1 rows.Source: D Validation Protocol and Measurement… · p. 15
  • The verifier samples from each row’s top-64 nucleus distribution, accepts nodes by hit or threshold, and chooses the deepest valid path, breaking ties by hits and path probability.If no path is valid, it selects the root; the server returns the chosen tokens, next token, and head inputs including hidden state, embedding, and KV entries.Source: D Validation Protocol and Measurement… · p. 15
  • In test runs, g includes prompt prefill; mobile-tree configurations run it separately before their first round, and Cloud+MTP token counts extend through its first stop token, as on the mobile device.During test runs, the mobile device kept its screen on at minimum brightness; temperature was not controlled or used to adjust reported full-run timings, and screen-off round trips took 14 ms longer. The lossless range was 58.7 for Cloud+MTP in an earlier session without the mobile device to 63.0 for exact ϵ=1 on HYCO’s tree across three sessions; the measured score penalty was ln 0.66 ≈ −0.42 over 7,513 rounds with exact verification and the MTP Drafter on the mobile device.Source: D Validation Protocol and Measurement… · pp. 15–16

E Quality per Task

Quality is assessed with question-only judge scores and separate reference-based diagnostics, with paired comparisons restricted to complete cases. The reported intervals reflect recorded-question variation, not variability across arbitrary deployments.

E Quality per Task

  • The SB judge scores first-turn answers using the question and answer, not a reference answer; two summarization prompts are consistently blocked, and HYCO–Cloud+MTP comparisons use 398 paired questions.Repeated comparisons involving SpecEdge-ϵ use the three-run complete-case intersection of 396 questions.Source: E Quality per Task · p. 16
  • The paired bootstrap intervals condition on recorded questions and do not estimate variation across arbitrary deployments; repeated quality averages available scores before complete-case pairing.The 15 task-wise contrasts are exploratory and not multiplicity-adjusted, and a confidence interval spanning zero does not establish equivalence.Source: E Quality per Task · p. 16, Table 7 · p. 17, Table 8 · p. 17
  • The QA question-only judge means are 49.1 for HYCO and 45.1 for Cloud+MTP across 80 questions, but they do not establish reference-answer correctness.On 43 reference-matched NQ-Open items, both systems score 30.2% under normalized reference containment; the paired difference is 0.0 percentage points (95% CI: −13.2 to +10.9).Source: E Quality per Task · p. 16
  • HYCO’s first-generation reference metrics are 95.0% GSM8K accuracy for both HYCO and Cloud+MTP, 65.7% versus 67.1% RAG accuracy, 19.2 versus 19.8 ROUGE-L, and 33.9 versus 33.7 BLEU.These are single-output descriptive checks; RAG measures normalized reference containment rather than full answer correctness.Source: E Quality per Task · p. 16

F Failure Cases

Observed cost exceptions and large judge-score changes are bounded evidence: the former occur in a few short RAG answers, while inspected answer-score discrepancies can reflect evaluator variation.

F Failure Cases

  • HYCO has 2, 1, and 0 non-saving cases across its three runs, all short RAG answers of approximately 40 tokens; this does not show output length alone causes every cost difference.The authors characterize these cases as an unfavorable operating condition.Source: F Failure Cases · p. 17
  • Among 25 screened large score changes, seven selected decreases include one especially low-scoring HYCO generation; an inspected identical answer text received scores of 15 and 95.The categories are author judgments, not independent adjudication or the overall error rate, and the example does not establish that all discrepancies are judge errors.Source: F Failure Cases · p. 17

G System Metrics and Reproducibility

System reporting separates server GPU efficiency from user-visible delivery and documents missing observations and latency boundaries. Additional measurements show SLM-drafter transfer to another model family, while moving that family’s head onto the device remains future work.

G System Metrics and Reproducibility

  • Table 11 separates server GPU time per output token from user-visible delivery, with mobile draft time excluding verification wait and phone prefill.For PD, subtracting verification wait also excludes drafting overlapped with that wait, so the mobile-time measure is not total active compute.Source: G System Metrics and Reproducibility · p. 17
  • The 32K HELMET evaluation contains 50 summaries per configuration, or 49 for PD, and compares percentage-point differences against Cloud+MTP using paired 95% CIs.Recall measures key-point coverage and precision measures support by the expert reference summary, not factuality against full novels.Source: Table 9 · p. 18
  • The SB PD run records 558 of 560 turns and its efficiency contrast uses 479 common questions; two turns lack assigned PRIME prefill, so reported PD GPU time may be slightly underestimated.On 32K, PD completes 49 of 50 requests, and its efficiency and quality contrasts use those 49 common questions.Source: G System Metrics and Reproducibility · pp. 17–18
  • HYCO’s mean/p90 TTFT is 846/1,922 ms on SB and 4,604/4,842 ms on 32K summaries; these are single-run traces, so first-token latency is separate from average delivery speed.Recorded SB network and serialization contribute roughly 40–58 ms per round, not as a standalone link-RTT measurement; other finalized configurations lack matching per-event traces.Source: G System Metrics and Reproducibility · p. 18

H Additional Cross-Family Measurements

  • On 480 SB questions with another verifier, server GPU, and engine, the single-pass 32-node hybrid tree reaches MAT 4.07, versus 3.16 for the four-pass 127-node SLM tree.These runs use answers up to 2,048 tokens and are not comparable with Table 2; the SLM Drafter transfers through a token bridge, while an on-device head for such families remains future work.Source: H Additional Cross-Family Measurements · p. 18

I Artifacts, Licenses and Compute

The section inventories artifact use and distributor-declared licenses, alongside compute records and the scope of its execution-efficiency metrics. It also identifies the study’s noncommercial research setting and AI assistance.

I Artifacts, Licenses and Compute

  • The study uses the artifacts for their intended purposes and conducts noncommercial research, with DPR passages licensed for noncommercial use and SpecEdge requiring a separate commercial-deployment license.The MTP Drafter is a derivative permitted by its Apache 2.0 license; test data are public and in English except for German translation source sentences.Source: I Artifacts, Licenses and Compute · pp. 18–19, Table 12 · p. 20
  • Table 11’s execution-efficiency metrics describe one-client execution, not multi-client serving throughput or priced service costs.It defines tokens per GPU-busy second as 1000/g per run and GPU-hours per million tokens as g/3.6 when g is in milliseconds.Source: Table 11 · p. 19
  • Final-run records total 5.04 GPU-busy hours and 31.00 cumulative mobile-request hours, excluding setup, idle time, unrecorded failed requests, calibration, validation sweeps, and additional cross-family experiments.Main Gemma deployment head-QAT used 100 steps of about 1.5 seconds each on one NVIDIA DGX Spark; the final GPU was a 96 GB RTX PRO 6000 in AWS g7e.2xlarge.Source: I Artifacts, Licenses and Compute · p. 19
  • An AI assistant helped edit the manuscript and write experiment and analysis scripts.Source: I Artifacts, Licenses and Compute · p. 19
  • Table 12 lists the artifacts and distributor-declared licenses, with UC-tr serving as the keep-set and 60 disjoint prompts for final head-QAT and activation calibration.The UC-tr use is specified in Appendix A.Source: I Artifacts, Licenses and Compute · p. 18, Table 12 · p. 20

Related papers

Loading 2610.21436v1…