Source-linked AI summary
Mix-Quant: Quantized Prefilling, Precise Decoding for Agentic LLMs
Haiquan Lu, Zigeng Chen, Gongfan Fang, Xinyin Ma, Xinchao Wang
TL;DR
Long-context, multi-turn agentic workflows make compute-intensive prefilling a major inference bottleneck, while quantizing the full pipeline can degrade performance through decoding error accumulation. Mix-Quant quantizes prefilling with NVFP4 while retaining BF16 decoding, largely preserving benchmark performance and achieving up to a 3× prefill speedup.
Problem
Repeated processing of long input contexts makes prefilling a major efficiency bottleneck, while full-pipeline low-bit quantization can degrade agentic task performance.
Method
Mix-Quant applies NVFP4 weight-and-activation quantization to prefilling while preserving BF16 precision for autoregressive decoding.
Results
Mix-Quant largely preserves performance across long-context and agentic benchmarks while achieving up to a 3× prefill speedup over BF16.
Takeaways & Limitations
Phase-aware quantization offers a favorable efficiency-performance trade-off for input-heavy LLM agentic inference.
Takeaways & Limitations
Quantized decoding can alter token predictions, causing sequence divergence and accumulating errors over time.
Abstract
from arXiv · showhide
LLM agents have recently emerged as a powerful paradigm for solving complex tasks through planning, tool use, memory retrieval, and multi-step interaction. However, these agentic workflows often introduce substantial input-side overhead, making the compute-intensive prefilling stage a key bottleneck in long-context, multi-turn inference. In this work, we propose Mix-Quant, a simple and effective phase-aware quantization framework for fast agentic inference. We first investigate FP4 quantization in agentic LLM workflows and observe that quantizing the entire inference process can incur significant performance degradation. In contrast, the prefilling stage exhibits substantial quantization redundancy and can therefore be quantized with minimal accuracy loss, despite being the dominant source of computation. Based on this insight, we apply high-throughput NVFP4 quantization to the prefilling phase while preserving BF16 precision for decoding. By decoupling prefilling acceleration from decoding quality, Mix-Quant combines phase-aware algorithmic quantization with hardware-efficient NVFP4 execution to alleviate the inference bottleneck in LLM agents. Extensive experiments across long-context and agentic benchmarks demonstrate that Mix-Quant largely preserves task performance while delivering significant efficiency improvements, achieving up to a 3x speedup during prefilling.
1 Introduction
Agentic workflows are input-heavy, making compute-intensive prefilling a major long-context inference bottleneck, while uniform quantization can degrade performance through decoding errors. Mix-Quant addresses this trade-off with NVFP4-prefilled computation and BF16 decoding, largely preserving task performance while achieving up to 3× prefill speedup.
- Motivation: Multi-step tool use, memory retrieval, and repeated inference loops create substantial context-processing overhead in LLM agent workflows.This overhead is especially important for long-context, multi-turn inference.
- Motivation: Aggressive compression applied uniformly across inference phases can cause non-negligible performance degradation, particularly when quantization errors accumulate during decoding.Weight-and-activation quantization reduces computational cost through low-bit matrix multiplications but can harm complex long-trajectory tasks.
- Motivation: Prefilling processes fixed contexts and is compute-intensive, whereas autoregressive decoding is more sensitive to accumulated numerical errors.These distinct roles make uniform quantization across both phases an unfavorable efficiency-performance trade-off.
- Mix-Quant: Mix-Quant applies high-throughput NVFP4 weight-and-activation quantization to prefilling while retaining BF16 precision for autoregressive decoding.This phase-aware design improves efficiency without introducing severe error accumulation during generation.
- Evaluation: 2–3× prefill speedup across varying sequence lengths and batch sizes is achieved while largely preserving task performance across long-context and agentic benchmarks.The comparison is against uniform NVFP4 quantization in the benchmark evaluation, while the contribution summary also reports up to a 3× prefill speedup over BF16.
2 Related Work
Related work establishes the foundations for agentic workflows, phase-disaggregated serving, and quantization-based LLM inference acceleration. These lines of research motivate treating prefilling and decoding differently because their computational profiles and quantization opportunities differ.
- Long-Context Agentic Workflows: Agentic LLM workflows combine reasoning with actions, external tools, memory, and feedback from interactive environments.ReAct interleaves reasoning traces with environment actions, Toolformer invokes external APIs, WebGPT performs browser-assisted question answering, and MemGPT explores memory management for long-lived interactions.
- Prefill-Decode Disaggregation: Prefill processes prompt tokens in parallel with large matrix multiplications, whereas decode autoregressively streams model weights and KV-cache entries.Serving systems exploit this distinction by separating prompt processing from token generation; Splitwise uses different machine configurations for the two phases, while DistServe disaggregates them.
- Quantization for LLM Inference: Weight-only quantization methods such as GPTQ and AWQ reduce memory traffic and suit bandwidth-bound decoding but provide limited speedup for long-context prefill.Their prefill benefit is limited because activations remain high precision and computation is not fully executed at low bit-widths.
- Quantization for LLM Inference: Weight-and-activation quantization can accelerate compute-bound prefilling, while aggressive W4A4 quantization may degrade full autoregressive inference quality.The supplied passage introduces this contrast but truncates before providing further details about full-model W4A4 behavior.
3 Method
Mix-Quant targets the dominant cost of long-context agentic inference by quantizing compute-intensive prefilling with NVFP4 while retaining BF16 precision for decoding. This phase-aware design exploits prefilling redundancy while avoiding the error accumulation and generation instability caused by quantized decoding.
- Motivation: Agentic workflows repeatedly append tools, observations, retrieved documents, actions, and intermediate states, rapidly expanding the input context and inference cost.Unlike conventional generation, agentic workflows typically produce only a few tokens per step while repeatedly processing much longer contexts.
- Motivation: Prefilling is compute-intensive because it performs parallel large-scale matrix multiplications across the full context, whereas decoding is sequential and often constrained by memory access and KV-cache I/O.This phase-level difference limits the benefit of weight-only quantization for prefilling, which remains dominated by dense computation and dequantization overhead.
- Quantization Rationale: Quantization errors are mainly representation-level perturbations during fixed-context prefilling but can accumulate during autoregressive decoding as altered tokens change the future history.In long agentic trajectories, such divergence can cause invalid tool calls, wrong actions, corrupted edits, or incorrect intermediate states.
- Quantization Rationale: In a 128K-context setting, the top-4096 tokens represent 3.125% of the context yet retain 95.8% of average attention mass across layers and heads.This concentration indicates substantial long-context redundancy and supports quantizing prefilling with limited impact on subsequent decoding.
- Mix-Quant Framework: Mix-Quant applies NVFP4 weight-and-activation quantization to prefilling and retains high-precision decoding to improve efficiency while stabilizing agentic generation.NVFP4 uses E2M1 FP4 values with local scales shared by groups of consecutive elements and targets Blackwell-generation low-precision tensor-core execution.
4 Experiments
Experiments across long-context, agentic, reasoning, and mathematical benchmarks show that Mix-Quant preserves performance better than uniform NVFP4 while accelerating prefill by nearly 3×. Phase-wise ablations attribute this advantage to quantizing prefill while retaining BF16 decoding.
- Benchmarks: Mix-Quant is evaluated on long-context, agentic, mathematical-reasoning, and interactive-memory benchmarks, including LongBench-V2, AA-LCR, BFCL v4, LongMemEval, τ2-bench, Math500, AIME24, and AIME25.The benchmark suite covers document understanding, synthesis, reasoning, tool use, long-term memory, and stateful conversations.
- Models and setup: Experiments compare original BF16, uniform NVFP4 W4A4, and Mix-Quant across Qwen3-8B, Gemma-4-26B-A4B-it, Gemma-4-31B-it, and Qwen3.5-9B.Models run on RTX 5090 and B200 GPUs using vLLM, with three independent runs reported by mean.
- Long-Context Agentic Benchmark Results: 42.85→38.64, 77.31→70.37, and 66.07→55.95 are the average-score drops for Qwen3-8B, Qwen3.5-9B, and Gemma-4-26B-A4B-it under uniform NVFP4.The results show that quantizing the entire inference process can substantially harm long-context agentic reasoning.
- Prefilling Stage Speedup: Nearly 3× average prefill speedup over BF16 is achieved across varying sequence lengths and batch sizes for Qwen3.5-9B and Qwen3-8B.The measurements use vLLM, FlashInfer attention, Blackwell NVFP4 W4A4 GEMM kernels, and identical configurations across methods.
- Phase-wise Quantization Ablation: Mix-Quant recovers uniform-NVFP4 losses, raising Qwen3-8B from 33.59 to 38.32 and Gemma-4-26B-A4B-it from 53.34 to 60.18.The corresponding BF16 baselines are 40.42 and 63.81, respectively.
- Phase-wise Quantization Ablation: Prefill-only FP4 quantization outperforms decode-only quantization, while BF16 decoding preserves generation quality under the phase-aware design.Uniform NVFP4 quantizes both stages; Mix-Quant quantizes only prefilling, and P16D4 quantizes only decoding.
5 Conclusion
The conclusion identifies a critical efficiency–performance dilemma in long-context agentic LLM inference: repeated long-context processing makes prefilling a major bottleneck, while full-pipeline low-bit quantization can degrade performance through autoregressive decoding errors.
- 5 Conclusion: Long-context agentic workflows repeatedly process large inputs, making the compute-intensive prefilling phase a major inference bottleneck.This overhead is central to the efficiency–performance dilemma identified in the conclusion.
- 5 Conclusion: Naively applying low-bit quantization across the full inference pipeline can degrade performance because errors accumulate during autoregressive decoding.The conclusion contrasts full-pipeline quantization with the need to preserve inference quality.
- 5 Conclusion: Mix-Quant is proposed as a phase-aware quantization framework addressing the efficiency–performance challenge in agentic LLM inference.The framework is introduced in response to the bottleneck and decoding-degradation dilemma.