PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision Is Not Overhead: One-Pass Block Drafting for Lossless Speculative Decoding in Vision-Language Models

Jungseob Lee, Seongtae Hong, Dongyub Jude Lee, Chanjun Park, Jaehyung Seo, Sugyeong Eo, Heuiseok Lim

arXiv:2609.00355v1cs.AIcs.CLcs.CV

TL;DR

VLM speculative decoding struggles because autoregressive drafters are kept small and often lose visual access, despite grounded image-conditioned text being highly predictable. GLANCE uses fused target states for one-pass block drafting and wide-tree verification, achieving up to 2.93× speedup while reproducing greedy decoding exactly on audited prompts. Its entropy-based gains favor grounded generation and identify free-running text as the method’s boundary.

  • Problem

    VLM speculative drafters remain sequential and often restrict image access, limiting drafting where grounded image-conditioned text is most predictable.

  • Method

    GLANCE uses a block-diffusion head over the target’s fused vision-language states to draft a whole block in one pass, then verifies a wide candidate tree in one target pass.

  • Results

    2.93× faster than autoregression on grounded workloads, GLANCE uses one draft pass per round versus eight for EAGLE3-VL and accepts 2.7× longer blocks than an equally trained EAGLE-3 head.

  • Takeaways & Limitations

    Accepted length follows target next-token entropy across tasks, with grounding improving draftability and the relationship transferring across targets and modalities.

  • Takeaways & Limitations

    Speculation falls below parity for single-letter ScienceQA answers, while free-running text remains a regime where autoregressive chains lead.

Abstract

from arXiv · show

Speculative decoding accelerates generation without changing its output, yet on vision-language models (VLMs) it has been caught in a self-defeating cycle. The drafter stays autoregressive, so it must stay small. A small drafter cannot afford the image at every step, so vision is compressed, pruned, or hidden. A drafter cut off from the image is then least reliable exactly where the image makes text predictable. We present GLANCE, the first one-pass block drafter that is lossless on an unmodified VLM target, and it breaks the cycle at both ends. A block-diffusion head reads the target's already-fused vision-language state, so vision costs the drafter nothing, and fills a whole block in one forward pass, so depth costs no sequential steps. A wide candidate tree is verified in one target pass, and every audited prompt reproduces greedy decoding exactly. Grounded workloads reward this most, entering a verbatim-copy regime whose long runs cost an autoregressive drafter a pass for every token and a block drafter one in total. Under one engine and one round budget, GLANCE decodes up to 2.93x faster than autoregression, from one draft pass a round where the production EAGLE3-VL head takes eight, and accepts 2.7x longer blocks than an EAGLE-3 head trained on the same corpus. One law organizes these results. Accepted length is set by the target's next-token entropy, with a fitted slope that steepens with grounding across all five tasks. The law transfers across targets and modalities and names its own boundary, since free-running text still favors a chain. Our code is available at https://github.com/js-lee-AI/GLANCE.

1 Introduction

VLM decoding is bottlenecked by the autoregressive loop, while existing VLM drafters make the drafter small by limiting visual access. GLANCE instead uses fused vision-language states and one-pass block drafting, yielding larger gains on grounded, low-entropy workloads.

  • Every generated token requires a target forward pass, making small-batch grounded generation memory-bandwidth bound by the decode loop.
  • Autoregressive VLM drafters require sequential passes with depth, while small-drafter constraints encourage compressing, pruning, or hiding image tokens.
  • Grounded answers often copy image text, producing low next-token entropy and longer draftable blocks than open-ended captioning.
  • Accepted length is governed by target next-token entropy, with grounding increasing the fitted slope across tasks.
  • GLANCE reads the target’s fused vision-language state and fills an entire block in one draft pass, then verifies a wide candidate tree in one target pass.
  • 2.93× faster than autoregression, GLANCE uses one draft pass per round where EAGLE3-VL uses eight, while preserving the target’s committed tokens.
  • 2.7× longer blocks are accepted than by an EAGLE-3 head trained on the same corpus and schedule.

2 Related Work

Prior speculative-drafting methods either retain sequential drafting or remove sequential bottlenecks through block-parallel approaches. GLANCE applies one-pass block drafting and tree verification to VLMs.

  • Speculative decoding commonly drafts from target hidden features using independent or sequential heads and candidate trees, with EAGLE-3 extending this line.
  • Block-parallel and diffusion-based methods address the drafter’s sequential bottleneck, including block-diffusion heads on frozen target hidden states.
  • Figure 2 depicts GLANCE’s VLM pipeline: fused vision-language states feed one-pass block drafting, prefix-tree construction, and one-pass ancestor-masked verification.

3 Preliminaries

Speculative decoding measures accepted tokens per verification round and uses candidate continuations to reduce target passes. GLANCE’s interface supplies all block-position marginals in one draft pass and organizes them into a verified prefix tree.

  • Autoregressive greedy decoding selects the target’s highest-probability token at each prefix, whereas speculative decoding verifies drafted continuations in rounds.
  • Acceptance length τ counts tokens committed per verification round, with τ=1 for autoregressive decoding and higher τ implying fewer target passes per emitted token.
  • The one-pass interface returns one marginal distribution at every block offset before any within-block token is committed.
  • A candidate tree is prefix-closed, verified in one packed target pass, and commits the longest prefix matching the target’s greedy continuation plus a bonus root token.
  • Drafting cost is independent of tree width in sequential passes, though increasing the tree budget adds verifier FLOPs within the single verification pass.

4 Method

GLANCE combines a frozen VLM target with a one-pass block head and wide-tree verification. Its verifier preserves greedy decoding under stated conditions, while the head inherits visual grounding from fused target states.

  • GLANCE drafts a whole future-token block in one head pass, then verifies a wide candidate tree in one target pass and commits the target’s greedy prefix.
  • The 1.05B-parameter head uses five fused target layers rather than raw visual tokens, inheriting image grounding after the target has merged vision into text representations.
  • Highest-scoring offset-wise prefixes form the candidate tree, whose nodes reproduce the target’s autoregressive conditionals during verification.
  • Moderate tree width improves accepted length without adding sequential draft passes, because the tree reuses the one-pass drafting cost.
  • Under unique target argmaxes and sufficient top-2 logit gaps, the greedy walk matches the frozen target’s autoregressive sequence bitwise in fp32.
  • A weak head can shorten accepted prefixes but cannot change the output; bf16 tie behavior requires a tie-aware gate that defers to the target.
  • The training setup freezes the vision tower and target decoder while training only the block head on target greedy generations.

5 Experiments

GLANCE is evaluated against VLM drafting baselines under exact-output and matched-training protocols. It gains speed and accepted length on grounded generation, while free-running text and short-answer classification define its practical boundaries.

  • Setup: The evaluation covers five task families from COCO captioning and TextVQA through InfographicVQA, DocVQA, and ChartQA, using Qwen3-VL-8B as the frozen primary target.Qwen3-VL-4B and Llama-3.2-11B-Vision appear only in the transfer study.
  • Setup: All greedy runs are gated for exact reproduction of the target’s decoding, with acceptance length measured independently of the serving engine.The protocol uses batch-one, decode-only generation capped at 256 new tokens; most tables let each system use its own operating point.
  • Head-to-head results: 2.93× faster than autoregression, GLANCE also runs up to 7.6% faster than the production EAGLE3-VL head on image-anchored tasks.Both speculative systems use the same engine, card, and 32-token round budget; EAGLE3-VL takes eight sequential draft passes while GLANCE takes one.
  • Task regimes: Free-running text favors the eight-pass chain, whereas grounded tasks favor GLANCE, with the split occurring at one prompt in 101 versus at least 70.The reported mechanism is that deep candidates drift without near-deterministic grounded text, while chains preserve coherence by construction.
  • Matched training: 2.7× longer blocks are accepted by GLANCE than by an EAGLE-3 head trained on the same corpus and schedule, pooled over five domains.The matched-training gap replicates on a second training corpus, with accepted lengths of 2.04 versus 1.29.
  • Losslessness: GLANCE reproduces greedy decoding on every audited prompt, while ViSpec and Medusa reproduce none under the tested relaxed acceptance rule.The audit is performed at fp32; bf16 can fail to reproduce the target even without a drafter.
  • Context length: At approximately 6.7K context, GLANCE and the production head reach parity on full-resolution InfographicVQA after GLANCE’s accepted length falls further.At 2K context, GLANCE is paired +5.8% faster while matching the head’s accepted length from one draft pass rather than eight.
  • Scope: Speculation falls below parity on ScienceQA’s single-letter answers, establishing the method’s scope as generation rather than classification.Under sampling, verified tokens remain drawn from the target’s distribution by construction, while byte-identity is undefined.

6 Why Grounded Is More Draftable

The paper models accepted block length as a function of the target’s root next-token entropy and tests how grounding changes that relationship. The law fits all five tasks, transfers across modalities, and has explicit tail and architecture boundaries.

  • Entropy law: Accepted length is modeled as a truncated-geometric mean whose match probability follows logit p_m(H) = b_0 − b_1H.Here H is root entropy, p_m(H) is the probability of matching the target’s greedy token, and b_0, b_1 > 0 are fitted parameters.
  • Entropy law: The fitted law has positive slopes across all five tasks, with grounding-associated slopes rising from 0.46 on captioning to 1.71 on templated ChartQA.The pooled fit over 7751 rounds has intercept b_0=0.96 and slope b_1=0.63; pooled R2 is 0.71.
  • Entropy law: Figure 4 plots root entropy H against expected accepted length E[a | H], showing every task-level fit between the AdaEDL lower bound and the 2 log N/H ceiling at N=31.The plotted points are measured entropy deciles, and the vertical axis uses a log scale.
  • Role of vision: Intervening on the image lengthens accepted blocks for every task and lowers DocVQA’s mean next-token entropy from 0.72 to 0.26 nats.The accepted-length increases are +3.5% on captioning, +6.8% on TextVQA, and +11.7% on DocVQA.
  • Transfer: The law transfers across targets and modalities, with positive fitted slopes on two additional VLM targets and on speech, chat, a non-Qwen backbone, and code.Reported slopes include 0.67 and 0.40 for additional VLM targets, 0.49 for speech, 0.64 for chat, 0.84 for a non-Qwen backbone, and 1.76 for code.
  • Boundary: The ordering is architecture-scoped: the method remains lossless on Llama-3.2-11B-Vision, but the characterization inverts on grounded short-answer domains.That target introduces vision through cross-attention rather than a token prefix.
  • Grounded tail: At the entropy floor, the single-hazard family underestimates the grounded tail: DocVQA’s measured floor is 5.23 versus the 3.41 pooled ceiling.Theorem 2 attributes the excess to a verbatim-copy fraction above an explicit threshold.
  • Grounded tail: Grounded tasks increasingly enter the copy regime, with rounds accepting at least eight tokens rising from 0.0% on captioning to 11.5% on templated ChartQA.A run of length ℓ costs an autoregressive drafter ℓ sequential passes but this one-pass drafter one.

7 Conclusion

GLANCE is a lossless one-pass block drafter for frozen vision-language models. Its gains come from inheriting fused vision context, filling depth in one pass, and harvesting grounded verbatim runs, while an entropy law identifies when the approach applies.

  • Method: GLANCE proposes whole blocks from the target’s fused states in one pass and uses a wide tree to increase accepted length.This replaces sequential drafting with parallel block filling.
  • Correctness: The method is bitwise identical to greedy autoregression for the evaluated frozen vision-language setting.
  • Scope: An entropy law captures when GLANCE pays off, transfers across drafter, target, and modality changes, and marks free-running text as a chain regime.

A Limitations

The guarantees and evaluations are bounded by batch-one decode-only conditions, greedy decoding, finite numerical precision, and sensitivity to near-ties. The bitwise result is therefore conditional rather than universal across batching, sampling, or numerical regimes.

  • Evaluation scope: All claims use batch-one, decode-only measurements, with no claim about larger batches.
  • Decoding regime: The bitwise guarantee applies to greedy decoding; sampled decoding preserves the target distribution by construction rather than byte identity.
  • Evaluation scope: Sample sizes are small and reported at each measurement in Appendices D and E.
  • Assumptions: Exact losslessness assumes unique target argmaxes whose logit gaps exceed packed-versus-unpacked perturbations.
  • Numerical boundary: In fp32, divergences can occur only at ties or margins below the perturbation threshold; bf16 uses a tie-aware gate to preserve the bound.

B.2 Derivation of the Law

The paper models accepted block length as decreasing with target next-token entropy, then extends the model to capture grounded, near-deterministic copying. Empirical fits support a grounding-ordered law while identifying finite-sample noise and a boundary where a two-state model is needed.

  • Entropy-to-match derivation: The slowly varying hazard approximation makes offset-wise survival factorize into marginal match probabilities, producing a truncated-geometric conditional mean.The resulting characterization is operational rather than an exact survival identity.
  • Entropy-to-match derivation: Entropy controls root matching: higher target entropy increases target top-1 error, while a logistic drafter-noise model yields an approximately affine logit match probability.Fano’s inequality supplies the monotonicity; the logistic and first-order affine forms are modeling steps.
  • Grounding and accepted length: Grounded and OCR generation has lower entropy and longer expected accepted blocks, with mean lengths 1.88, 2.12, 3.08, 2.67, and 3.71 across the five tasks.The ordering is captioning < TextVQA < DocVQA < InfoVQA < ChartQA under the reported task groupings, with templated prompts noted for ChartQA.
  • Grounded tail and limits: Entropy-only prediction has an intrinsic noise ceiling because within-round accepted-length variance can dominate predictable between-round variation.The measured law realizes 89%/63%/39% of the entropy-only ceilings on captioning/TextVQA/DocVQA, plus 62% on InfoVQA and 50% on ChartQA.
  • Grounded tail and limits: A two-state determinate-or-uncertain survival model captures the grounded low-entropy tail beyond the geometric cap, although both the geometric and two-state fits face absolute rejection on small samples.On prompt-disjoint held-out tests, the two-state model has lower mean χ2 than the geometric model, 4.4 versus 11.9, without claiming exact data generation.

C Classic Two-Model Speculative Decoding

Classic two-model speculative decoding accepts substantial blocks with image-conditioned drafts, but its drafting cost makes it slower than autoregression on every task. Removing image access sharply reduces acceptance, showing that visual conditioning is central to whether classic drafting pays on grounded VLM workloads.

  • Classic two-model baseline: The image-conditioned 4B draft accepts τ=3.53–4.47 but achieves only 0.61–0.80× speedup, remaining slower than autoregressive decoding.A half-size draft costs roughly half a target forward for each drafted token, so acceptance does not amortize its cost.
  • Classic two-model baseline: The image-conditioned 2B draft accepts τ=3.11–3.97 and remains below 1× speedup throughout, at 0.66–0.90×.Reducing draft size narrows the deficit but does not close it.
  • Classic two-model baseline: Classic two-model speculation is a net slowdown on all five tasks even though it leads acceptance on four, because wall-clock depends on draft cost per accepted token.Speedup also saturates beyond a moderate budget, making added width increasingly ineffective.

D Implementation Details

GLANCE uses a frozen-target block head, one-pass tree drafting, and task-specific evaluation protocols designed to preserve losslessness and make within-engine comparisons interpretable. The implementation spans matched training, multiple targets, OCR adaptation, and controlled timing and acceptance measurements.

  • Extensions and targets: The implementation includes OCR retraining with 10.2K self-generated OCR rows and adapters for Qwen3-VL-4B, Qwen2.5-VL-7B, and Llama-3.2-11B-Vision.The OCR checkpoint is a partial-epoch warm-started model used to demonstrate selective grounded-task lift.
  • Training comparisons: On a matched three-task probe, the budget-63 GLANCE tree reaches acceptance ratio 2.53 and 2.73 across all five domains versus the depth-3 EAGLE3 chain’s 1.60.The advantage persists across probe sizes, while depth-7 adds at most 0.01.
  • Evaluation protocol: The evaluation fixes batch one, greedy decoding, a 256-token cap, and byte-identity gating, while reporting only within-engine speedup ratios.Cross-engine absolute milliseconds are avoided because hardware and runner differences rescale the arms differently.

E Timing Ledger

The timing ledger shows that decode-loop costs dominate visual prefill, while GLANCE’s one-pass drafting yields faster grounded decoding under matched conditions. Its advantage is tied to reduced sequential draft passes and remains sensitive to workload regime.

  • Cost distribution: 3–14% of a token’s end-to-end cost comes from visual encoder prefill, leaving the decode loop as the dominant cost.The prefill share decreases as generated outputs grow.
  • Workload regimes: +6.3% is the geometric-mean GLANCE advantage across grounded tasks, versus −17.7% across the two open tasks.The overall geometric mean is −4.0%.
  • Tree and chain: 1.36× is the zero-confound speedup for the identical head run as a tree rather than a chain.The comparison isolates tree width in the same harness.
  • Draft-pass cost: 24.8 ms plus 1.82 ms per sequential draft pass describes production-head round time across five configurations.The fitted constant term corresponds to 1.03 target forwards.

F Additional Transfer and Robustness Results

Additional analyses support entropy as the transferable organizing variable, show that matched visual conditioning improves drafting, and identify boundaries where the method’s advantage changes or reverses. Robustness checks cover engines, targets, modalities, verifier budgets, context, and training data.

  • Entropy-law transfer: 0.979 and 0.991 are the isotonic-curve R2 values for InfoVQA and ChartQA, respectively.Both tasks extend the five-task entropy-acceptance family, with ChartQA carrying the steepest slope.
  • Grounded-tail result: 4.42 against 3.22 and 6.92 against 4.50 are the measured floors and caps for the two remaining tasks.Both floors exceed their corresponding caps, extending the copy-regime excess beyond DocVQA.
  • Image intervention: +3.5%, +6.8%, and +11.7% are the accepted-block gains from the correct image on captioning, TextVQA, and DocVQA.Every interval excludes zero, and the effect scales with grounding.
  • Image intervention: 0.26 against 0.72 nats is DocVQA’s target mean next-token entropy with the correct versus mismatched image.After controlling for entropy, the pooled residual visual effect is non-positive at −0.07.
  • Transfer: All five pre-registered transfer conditions pass, supporting the empirical law across drafter and target axes.Panel A varies the drafter on a fixed target, while Panel B varies the target.
  • Cross-modality transfer: b1 > 0 across speech recognition, code, chat, and a non-Qwen backbone supports functional-form transfer beyond vision.Grounded VL and code follow a determinate-copy route, while speech behaves like a uniform position-level lift.
Loading 2609.00355v1…