PaperlayerView exact revision on arXiv

Source-linked AI summary

Raincheck: Deferred Multimodal Prefill for Efficient Multi-Turn MLLM Inference

Wenjie Huang, Jun Zhang, Hongyaoxing Gu, Haoyang Huang, Tianqi Xu, Xinyue Hao, Xinyi Hu, Xinle Du, Jun Dai, Shuang Ge, Chen Lei, Yue Li, Mingcheng Wan

arXiv:2610.39562v1cs.CLcs.AI

TL;DR

Long multimodal inputs increase time to first token, while hard pruning can discard information later questions need. Raincheck defers unselected tokens for prefill during decode and after answer return. Across two models and four pruning methods, it recovers quality lost to matched hard pruning while reducing MultiVerse first-turn latency with little TPOT increase.

  • Problem

    Long multimodal inputs increase time to first token, and hard pruning can permanently exclude information that later questions may require.

    Source: §1 · pp. 1–2

  • Method

    Raincheck prefills selected multimodal tokens first and progressively restores unselected tokens during decode and after answer return, using attention visibility control and affine state mapping for context updates.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    Across two models and four pruning methods, Raincheck recovers on average 77.7% and 60.4% of pruning-induced quality loss on MultiVerse and ConvBench, respectively.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Raincheck improves aggregate quality over matched hard pruning across image and video dialogues on both evaluated models.

    Source: §5 · p. 10

    The evaluation is single-request, establishing the opportunity for latency-sensitive inference in that setting.

    Source: §5 · p. 10

Abstract

from arXiv · show

Multi-turn conversations with multimodal large language models (MLLMs) reuse input context, but long multimodal inputs increase time to first token (TTFT). Query-aware hard pruning reduces prefill tokens but permanently excludes information that later questions may require. We propose Raincheck, a framework for deferred multimodal prefill that decouples initial token selection from later information availability. An existing pruning method selects tokens for initial prefill; Raincheck preserves and progressively prefills unselected tokens in source order using unused compute capacity during decode. Unfinished prefill continues after answer return during available scheduling gaps. Attention visibility control and affine state mapping enable context updates after generation begins in full-attention and hybrid recurrent models. Prefill-decode fusion reduces repeated weight reads. We evaluate Qwen3-VL-8B and Qwen3.5-9B with 30% initial multimodal-token retention. Across both models and four pruning methods, Raincheck recovers on average 77.7% and 60.4% of the quality lost to matched hard pruning on MultiVerse and ConvBench, respectively. On MultiVerse, the SGLang implementation reduces mean first-turn TTFT by up to 24.1% relative to full prefill, with less than 0.5% increase in mean time per output token (TPOT).

1 Introduction

Long multimodal inputs raise time to first token, while hard pruning can discard information needed for later questions. Raincheck prefills selected tokens first and restores deferred tokens as compute becomes available, recovering quality while reducing first-turn latency.

1 Introduction

  • Raincheck prefills selected multimodal tokens plus all text and control tokens first, then processes deferred tokens in source order during decode and after answer return.Unused compute during decode and intervals between requests provide opportunities to continue prefill; a per-step token limit controls TPOT.Source: §1 · pp. 1–2
  • Attention visibility control and online affine state mapping let full-attention and hybrid recurrent models update context after generation starts, while fused execution reduces repeated weight access.For recurrent layers, recorded transitions for trailing text and generated tokens support the corrected state without rerunning the full sequence.Source: §1 · p. 2
  • Query-aware pruning can shorten TTFT but permanently discards unselected tokens that may matter to ongoing answers or later questions.Raincheck addresses this mismatch by recovering deferred tokens during decode and after answer return.Source: §1 · pp. 1–2
  • Across two models and four pruning methods, Raincheck recovers on average 77.7% and 60.4% of pruning-induced quality loss on MultiVerse and ConvBench.On MultiVerse, mean first-turn TTFT falls by up to 24.1% and mean TPOT rises by less than 0.5% relative to full prefill.Source: §1 · p. 2

2 Raincheck

Raincheck separates fast initial response from later recovery of omitted multimodal input, making completed tokens available to ongoing generation and later turns. It combines deferred prefill with attention control, recurrent-state correction, and fused execution.

2 Raincheck

  • Raincheck extends input context after generation begins while keeping deferred input independent of the already processed suffix.Full-attention visibility and recurrent state mapping support context updates; fused execution limits repeated weight access.Source: §2 · p. 3

2.1 Design Motivation

  • At 30% retention, first-question MMTok selection omitted an average 19.34% of second-question selections on Qwen3-VL-8B and 17.32% on Qwen3.5-9B.The section argues selection should determine when tokens are processed, not whether they remain available.Source: §2.1 · p. 4
  • At small batch sizes, decode is often memory-bound, leaving compute capacity for deferred prefill; the token budget controls TPOT increases.With 31 deferred tokens per step, TPOT rose from 6.15 to 7.33 ms on Qwen3-VL and from 6.43 to 7.88 ms on Qwen3.5.Source: §2.1 · p. 4

2.2 Deferred Multimodal Prefill

  • An existing pruning method selects important tokens for initial prefill, while Raincheck preserves the unselected tokens in source order for later processing.Initial prefill includes selected multimodal tokens and all text, control, and special tokens; deferred tokens are prefetched in consecutive chunks during decode, and completed chunks become available to generation and later turns.Source: §2.2 · p. 5

2.3 Context Updates during Generation

  • Deferred chunks use causal attention over the processed prefix while excluding intervening input and generated tokens; decode sees processed input and generated history.For GDN layers, Raincheck maps the prefix-state change through recorded suffix transitions to correct the live state without rerunning the suffix.Source: §2.3 · pp. 5–6

2.4 Execution and Completion

  • The co-located executor fuses a deferred chunk with the decode token to share weight access, and unfinished prefill continues in scheduler gaps after answer return.The evaluated policy completes deferred tokens before a later turn reuses the context; an early follow-up waits for completion.Source: §2.4 · p. 6

3 Experiments

The experiments test whether deferred prefill recovers quality under matched initial token selections, reduces first-turn latency with a controlled TPOT increase, and retains gains across retention ratios and chunk sizes.

3.1 Experimental Setup

  • Evaluations compare Full, Hard, and Ours on Qwen3-VL-8B and Qwen3.5-9B at 30% initial retention, adapting four pruning methods with matched Hard/Ours settings.The datasets include multi-turn image and video dialogue benchmarks; main latency comparisons use SGLang on one H20 GPU per worker with one running request.Source: §3.1 · p. 7

3.2 Multi-Turn Quality Recovery

  • 77.7% of the Full-to-Hard score loss was recovered on average across eight model and pruning-method pairs on MultiVerse.With MMTok, aggregate quality rose from 48.77 to 55.22 on Qwen3-VL-8B and from 37.53 to 43.12 on Qwen3.5-9B.Source: §3.2 · p. 7, Table 2 · p. 8
  • 0.75 to 8.26 points of improvement over Hard appeared in the ConvBench mean across all eight model and pruning-method pairs.With MMTok, scores rose from 38.10 to 41.83 on Qwen3-VL-8B and from 35.53 to 43.79 on Qwen3.5-9B.Source: §3.2 · p. 8

Multi-turn video quality.

3.3 Inference Efficiency

  • 24.1% and 17.9% lower T1 TTFT on MultiVerse came with mean TPOT increases of 0.13% and 0.47% on Qwen3-VL-8B and Qwen3.5-9B, respectively.On MT-Video-Bench, mean E2E latency decreased by 4.83% and 6.56% relative to Full.Source: Table 4 · p. 8, §3.3 · pp. 8–9
  • All MultiVerse requests finished restoration within the measured 512-token traces, while 41.67% and 68.75% of video observations needed post-response completion.The average video post-response interval was 74.0 and 162.8 ms over all requests, including those already restored.Source: Table 5 · p. 9, §3.3 · p. 9

3.4 Ablation Studies

  • Restoration improved aggregate quality at every tested initial retention ratio, though lower retention generally left more score loss to recover.In the Qwen3-VL-8B video sweep, recovery was nonmonotonic: 50% retention yielded a larger gain than 30%.Source: §3.4 · p. 9, Figure 9 · p. 10
  • Quality gains persisted across chunk sizes 7, 15, 31, and 63 at 30% retention; MultiVerse improved over Hard in all eight comparisons by 2.73 to 6.23 points.Across the plotted settings, MT-Video-Bench scores spanned 0.54 points on Qwen3-VL-8B and 0.85 points on Qwen3.5-9B.Source: §3.4 · p. 9, Figure 9 · p. 10

4 Related Work

Prior multimodal token-pruning work reduces computation through different selection and compression strategies; Raincheck instead adds deferred input states during generation and after the answer.

4 Related Work

  • Pruning methods operate in modality encoders, between model layers, or during generation, using saliency, diversity, or query relevance to select tokens.Query-aware selection may omit information useful for an ongoing answer or later questions.Source: §4 · p. 10
  • Some approaches preserve pruned information by aggregating token features or learning compressed representations.Other methods retain candidates or intermediate states for later retrieval, reselection, or on-demand computation.Source: §4 · p. 10
  • Raincheck separates initial selection from recovery by adding deferred input states during generation and after answer return for later reuse.Attention visibility control and recurrent state mapping support insertion before an already processed suffix.Source: §4 · p. 10

5 Conclusion

Raincheck addresses the tension between fast responses and retaining input context. It recovers deferred tokens during and after generation, improving aggregate quality over matched hard pruning while reducing MultiVerse first-turn TTFT.

5 Conclusion

  • Raincheck processes selected tokens up front and recovers deferred tokens during generation and after answer return.The framework targets faster answers without permanently losing the input skipped during initial prefill.Source: §5 · p. 10
  • Attention visibility control and affine state mapping make recovered context usable for subsequent decoding, while fused execution shares weight access between prefill and decode.These mechanisms support context updates after generation begins in the evaluated models.Source: §5 · p. 10
  • Across image and video dialogues on both evaluated models, Raincheck improves aggregate quality over matched hard pruning.On MultiVerse, mean first-turn TTFT falls by up to 24.1% relative to full prefill, with less than a 0.5% increase in mean TPOT.Source: §5 · p. 10
  • The evaluation uses a single request, leaving concurrency and state-mapping assumptions as open questions.The conclusion frames the demonstrated opportunity as latency-sensitive inference.Source: §5 · p. 10

AI Use Statement

The authors disclose using generative AI tools for language editing and code assistance and take responsibility for the work.

AI Use Statement

A Notation and Implementation Details

This appendix specifies Raincheck’s notation, sequence and cache conventions, execution details, and recurrent-state mapping. It also describes the approximation boundary: state joins are exact for recorded factors, while a full rerun can produce different factors.

A.1 Notation and Figure Conventions

  • Table 6 collects notation and tensor shapes; the figures distinguish important, deferred, intervening, and generated tokens and show recurrent states as sequence endpoints.Earlier context is included in the prefix checkpoint, and the illustrated examples repair the state after y1 before processing y2.Source: Table 6 · p. 15, §A.1 · pp. 15, 17

A.2 Sequence and Checkpoint Conventions

  • Deferred tokens retain source order and are inserted before generated tokens; attention visibility and cache availability follow processed regions, not original token positions.The evaluated insertion boundary has no intervening input, and original positions supply position encodings while physical regions determine storage.Source: §A.2 · pp. 17–18

A.3 Execution Details

  • Both backends fuse deferred-token and decode processing, while scheduling chunks and completion policies govern recovery without replaying the full sequence.The scheduler uses captured shapes and a configurable budget; a 16-token decode guard preserves the initially pruned context before deferred prefill begins.Source: §A.3 · pp. 18–19

A.4 Memory Accounting

  • Deferred prefill changes when input states are computed, not the final cached-token count under full recovery; reserved pages do not provide Hard’s persistent KV-capacity savings.Peak memory also depends on temporary activations and serving-engine cache pools and graph buffers, not component counts alone.Source: §A.4 · pp. 19–20, Table 7 · p. 19

B.1 Single-Token State Update

  • For each token, the recurrent state update is affine in the incoming state, combining a state transformation with an additive key-value contribution.The augmented matrix representation derives affine composition, while the implementation maintains the recurrent state and transition coefficients directly.Source: §B.1 · p. 20

B.2 Suffix Mapping and State Join

  • Recorded suffix transitions compose into an affine map whose linear coefficient propagates the incoming state and whose additive coefficient accumulates token updates.The state-difference identity cancels the shared additive contribution, allowing incremental correction when the prefix state changes; online decoding extends the transition matrix with each generated token.Source: §B.2 · pp. 20–21, §B.3 · p. 21

B.4 Short-Convolution Boundary Handling

  • For input-end insertion, deferred tokens can change convolution history at the start of generation, so repair recomputes a bounded leading segment and avoids replaying the growing suffix.The state-difference identity is exact for recorded factors in the remaining suffix.Source: §B.4 · p. 21

B.5 Approximation Relative to Recomputed Factors

  • The fixed-factor join is exact, but a full rerun under enlarged context can change suffix factors; lower-layer changes may alter higher-layer keys, values, and gates.The comparison holds suffix tokens fixed, whereas autonomous generation can also change the token sequence.Source: §B.4 · p. 21, §B.5 · p. 22

B.6 State-Map Scaling and Numerical Validation

The section evaluates the online GDN state-map join against direct tokenwise application of identical recorded factors, then reports its incremental timing and storage costs.

B.6 State-Map Scaling and Numerical Validation

  • The diagnostic compares a dense state join with direct tokenwise application of identical recorded factors on nested prefixes of a fixed 4,096-token BF16 stream.It uses one Qwen3.5-shaped GDN layer and excludes four convolution-sensitive prefix tokens; error measures implementation differences under the recorded factors.Source: §B.6 · p. 22
  • 2.02–2.12 µs per token is the incremental online-step time across four suffix lengths, while one join takes 10.60–10.81 µs.Timings are CUDA-graph medians of nine samples, with Python dispatch excluded.Source: §B.6 · p. 22
  • Map storage is 2 MiB per layer, or 48 MiB across 24 GDN layers, within the 192 MiB auxiliary pool.The per-layer map uses one 128 × 128 FP32 matrix for each of 32 value heads, independent of suffix length; totals exclude weights and ordinary KV storage.Source: §B.6 · p. 22
  • The state-map diagnostic measures one layer’s recorded-factor join, whereas Table 31 measures serving TPOT in the fusion control.The reported join timings should not be read as serving-level TPOT measurements.Source: §B.6 · p. 22, Table 8 · p. 23

C.1 Benchmarks and Aggregation

This block describes the benchmark populations, scoring and serving protocols, and quality evaluations. Raincheck improves several reported quality comparisons, while the scoring scope and evaluation procedures differ across benchmarks.

C.1 Benchmarks and Aggregation

  • ConvBench’s main score evaluates each current-turn answer rather than complete-dialogue consistency, while MT-Video-Bench equally weights its six category scores.The benchmark summary table covers task, evaluated population, and scoring metric.Source: §C.1 · p. 23, Table 9 · p. 23

C.2 Quality Generation and Scoring

  • The quality protocol fixes generation budgets across Full, Hard, and Ours, and uses benchmark-specific answer evaluators, including structured fallbacks for unresolved MMBench responses.The MultiVerse score is a paper-defined proxy because its public release omits evaluation code; it combines checklist coverage and answer quality.Source: §C.2 · p. 24

C.3 Video Sampling and Evaluation

  • MT-Video-Bench averages checklist scores within turns, dialogues, and categories before equally weighting six categories; its T1, T2, and T3 groups cover 2, 3, and 5 annotated categories.Overall covers 2,078 annotated answers across up to eight turns, and category coverage prevents direct comparison of score levels across turns.Source: §C.3 · p. 25

C.4 Generation, Serving, and Judge Profiles

  • SGLang timing uses one H20-3e GPU, tensor parallelism one, and one resident request, with separate repetitions for image-dialogue, restoration-interval, and video measurements.The profile uses FlashAttention-3 and CUDA graphs; video timing uses 64 frames and the slow native video processor.Source: §C.4 · p. 26

C.5 Judge Instructions and Scoring Examples

  • The dialogue judges receive benchmark-specific evidence: MultiVerse gets the image and causal history, ConvBench gets the caption and current-turn answers, and MT-Video-Bench gets the answer checklist.This makes the judge’s available history match the evaluation target: causal dialogue quality, local-turn quality, or answer-level checklist coverage.Source: §C.5 · p. 26

D.1 Single-Turn Quality

  • Raincheck improves MathVerse over Hard for all four pruning methods on both models, with mean gains of 1.50 points on Qwen3-VL-8B and 8.29 on Qwen3.5-9B.On Qwen3.5-9B, Ours scores range from 63.55 to 64.67, against a Full score of 64.34; other reported single-turn differences are small.Source: §D.1 · p. 27, Table 11 · p. 28

D.2 Context Available during Decode

  • Hard stays at the initial partition, while Raincheck progressively expands the decode-active context from a 30.07% mean initial retention across five single-turn datasets.The reported table distinguishes initial retention from the mean fraction of multimodal tokens available during decoding.Source: §D.2 · p. 28, Table 12 · p. 28

D.3 Quality Differences and Statistical Uncertainty

  • 31-token Raincheck scores are 1.42 points below Full on Qwen3-VL-8B and 0.15 points below Full on Qwen3.5-9B for MT-Video-Bench overall.The same evaluation reports positive overall gains over Hard for both models; category gains are mixed, so aggregate recovery does not imply improvement on every question type.Source: §D.3 · p. 29

D.5 Retention and Chunk-Size Sensitivity

The section tests retention and restoration-chunk settings, then compares restoration during generation with restoration only after the first answer. Quality gains vary by benchmark and setting.

D.5 Retention and Chunk-Size Sensitivity

  • The 31-token configuration’s point estimates exceed Hard across tested initial-retention settings on MultiVerse, ConvBench, and MT-Video-Bench.The MultiVerse sweep tests 30%, 50%, and 70% retention; the ConvBench and video sweeps extend the comparison to the second image benchmark and multi-turn video dialogue.Source: §D.5 · p. 30, Table 17 · p. 31, Table 18 · p. 31
  • Across 16 MultiVerse and ConvBench chunk comparisons, all Ours-minus-Hard point estimates are positive and 14 paired 95% intervals exclude zero.Image-dialogue scores are not monotonic in chunk size; on Qwen3.5-9B video, the 64-token setting scores up to 1.32 points above tested alternatives.Source: §D.5 · p. 31, Table 20 · p. 32, Table 21 · p. 32
  • ConvBench context-adherence and complete-dialogue preference estimates are all positive, but several intervals include zero; presentation-order disagreement is 70.02% and 69.14%, respectively.The comparisons permit ties, and zero denotes equal preference; the endpoints are supporting dialogue evidence separate from local-turn quality.Source: §D.5 · pp. 31–32, Table 23 · p. 33

D.6 Quality Effects of Restoration Timing

  • The MultiVerse timing experiment compares Ours with Post-answer on 647 dialogues per model, using 30% initial retention, 31-token chunks, and one chunk per decode step.Post-answer keeps the pruned context through the first answer and restores deferred tokens before the next turn; Ours starts restoration after a 16-token decode guard.Source: §D.6 · p. 33
  • 4.93 and 5.70 points are the all-turn MultiVerse gains of Ours over Post-answer on Qwen3-VL-8B and Qwen3.5-9B, respectively.First-turn gains are 13.51 and 16.51 points; follow-up gains are 2.00 and 1.99 points, though third- and fourth-turn intervals include zero.Source: §D.6 · p. 33, Table 24 · p. 33

E.1 SGLang Fixed-Trace Protocol and Complete Results

The SGLang experiments measure paired latency under fixed workloads, then examine completion after answer return, scheduling choices, query-dependent selection, and deferred-token budgets. Results show near-Full mean TPOT across measured turns, while follow-up timing, chunk size, and fusion reveal important operating trade-offs.

E.1 SGLang Fixed-Trace Protocol and Complete Results

  • 0.010% on Qwen3-VL and 0.157% on Qwen3.5 are Raincheck’s all-turn mean TPOT changes relative to Full; deferred prefill completes during every T1 request.The completed prefix is reused on later turns without repeating deferred prefill.Source: §E.1 · p. 34

E.2 Completion after Answer Return

  • 73.61 and 162.12 ms are Ours’ mean T2 TTFT increases relative to Full for the two models; mean T3+ TTFT is 18.00 versus 18.01 ms and 34.30 versus 34.35 ms.The T2 delay arises when deferred tokens remain unmaterialized after the first answer; the completed answer is released before the parent prefix is ready.Source: §E.2 · p. 37

E.3 SGLang Scheduling and Fusion Controls

  • 5.916% on Qwen3-VL and 12.932% on Qwen3.5 are mean TPOT reductions for fused versus serial execution; increasing c from 7 to 31 lowers Qwen3.5’s TPOT increase from 2.669% to 0.381%.The fusion comparison includes its associated scheduling change; c = 63 increases mean TPOT on both models relative to c = 31.Source: §E.3 · pp. 37–38, Table 29 · p. 38, Table 31 · p. 38

F.1 Query-Dependent Token Selection

  • Answer-conditioned selections differ from question-only selections, and the diagnostic measures selection changes without testing whether newly selected tokens are necessary to answer.The comparison uses unchanged Full answers; answer length correlates positively with selection difference, but high-quality subsets do not isolate answer length’s effect.Source: §F.1 · p. 39, Table 32 · p. 39

F.2 Deferred-Token Budget

  • Smaller deferred-token budgets cover more output positions, tracing restoration trajectories with growing context rather than fixed-context kernel timings.Measurements average intervals processing full-size chunks; smaller terminal chunks and post-response completion are excluded.Source: §F.2 · pp. 39–40

F.3 Inter-Turn Completion Windows

The paper measures intervals between multimodal user messages and individual tool-call durations as possible windows for deferred prefill. These timing populations are defined separately, and completion before the next request depends on the remaining tokens and available scheduler capacity.

F.3 Inter-Turn Completion Windows

  • Among 102 cleaned ShareChat observations, intervals run from preceding assistant-message creation to next user-message creation and may include answer-generation time; TraceLab positive tool-call latency has median 0.3 s.Source: §F.3 · p. 40
  • Whether deferred prefill finishes within an application interval depends on the deferred remainder and available scheduler capacity; an early follow-up waits at the completion barrier.Source: §F.3 · p. 40

G.1 Evaluation Scope

The evaluation establishes a batch-size-one systems trade-off on Qwen3 full-attention and Qwen3.5 GDN, while other serving settings require separate evaluation. The paper also discusses how disaggregated recovery, pruning approaches, and information-retention methods relate to Raincheck.

G.1 Evaluation Scope

  • The systems measurements establish the co-located execution trade-off at batch size one; other recurrent architectures, batched serving, and speculative decoding require separate evaluation.The quality evaluation uses a 512-token per-turn budget and dataset-specific judges.Source: §G.1 · pp. 40–41

G.2 PD-Disaggregated State Join

  • Turn-boundary recovery lets later turns use recovered context, but the first answer receives no quality benefit from deferred-token recovery and an early follow-up can incur a recovery delay.Separate workers also require prefill capacity and state transfer; the final join and boundary repair remain on the decode side.Source: §G.2 · p. 41

H.1 Multimodal Token Pruning

  • Multimodal pruning methods select tokens using criteria including diversity, coverage, representation change, query relevance, and modality budgets.Source: §H.1 · p. 42

H.2 Information Retention and Recovery

  • Raincheck schedules deferred multimodal tokens for source-order prefill, progressively expanding context available to the ongoing answer; evaluated policies complete the input prefix before later-turn reuse.Source: §H.2 · p. 42

H.3 Execution and Hybrid Architectures

Raincheck co-schedules prefill and decode within a request while maintaining state dependencies for deferred tokens. This requires updates to recurrent state as well as attention visibility in hybrid models.

H.3 Execution and Hybrid Architectures

  • Raincheck co-schedules prefill and decode within the same request and maintains state dependencies introduced by deferred input tokens.Sarathi fuses prefill and decode linear computations to share weight accesses, whereas Raincheck’s stated distinction is same-request co-scheduling and state dependencies.Source: §H.3 · p. 42
  • Hybrid models such as Qwen3.5 combine full attention with GDN layers, so deferred input requires updating recurrent state as well as attention visibility.Source: §H.3 · p. 42
Loading 2610.39562v1…