PaperlayerView exact revision on arXiv

Source-linked AI summary

RAMP: Recurrent Adaptive Mixed Precision for Efficient Prefill in Recurrent-Depth Language Models

Ke Wan, Chen Chen

arXiv:2610.13041v1cs.CL

TL;DR

Recurrent-depth models incur added inference cost, especially during long-context prefill, and repeated quantization perturbs their evolving representations. The paper analyzes error propagation and introduces RAMP, which quantizes early refinement and restores high precision for terminal steps. Across contexts up to 4K, RAMP preserves near-full-precision prefill quality while reaching up to 1.16× first-prefill speedup on an NVIDIA H100, a 13.9% latency reduction.

  • Problem

    Long-context prefill repeatedly processes the full input across recurrent steps, and uniform weight-and-activation quantization introduces numerical perturbations at each step; the paper asks how to allocate precision across depth while preserving fidelity.

    Source: §1 · p. 1

  • Method

    RAMP analyzes recurrent quantization-error propagation and uses low-precision weight-and-activation computation early, restoring BF16 precision for terminal refinement.

    Source: §1 · p. 2, §7 · p. 12

  • Results

    Across contexts up to 4K, RAMP preserves near-full-precision prefill quality and achieves up to 1.16× end-to-end first-prefill speedup on an NVIDIA H100, a 13.9% latency reduction.

    Source: §1 · p. 2, §7 · p. 12

  • Takeaways & Limitations

    Under locally contractive recurrent dynamics, earlier quantization errors receive more subsequent attenuation, and fixed-budget experiments support placing quantized computation earlier in recurrent depth for better fidelity.

    Source: §7 · pp. 11–12

    The error analysis assumes local contraction within the operating region relevant to quantized inference, rather than global contraction of the recurrent operator.

    Source: §3 · p. 4

Abstract

from arXiv · show

Recurrent-depth language models improve inference-time reasoning by repeatedly applying a shared network to refine internal representations, but the additional recurrent computation also increases inference cost. This overhead is particularly pronounced during long-context prefill, where the full input sequence is repeatedly processed across recurrent depth. Weight-and-activation quantization offers a natural way to accelerate this computation through lower-precision arithmetic, yet repeatedly applying quantized computation introduces numerical perturbations that accumulate along the recurrent trajectory. We systematically analyze how such errors propagate across recurrent depth and show that quantization errors introduced earlier are attenuated by more subsequent recurrent steps and therefore have a smaller effect on the final representation. Matched-budget experiments strongly support this prediction, showing that placing quantized computation earlier in recurrent depth consistently yields the best fidelity. Based on this insight, we introduce RAMP (Recurrent Adaptive Mixed Precision), which performs early recurrent refinement using low-precision weight-and-activation computation and restores high precision for terminal refinement. Across context lengths up to 4K, RAMP preserves near-full-precision prefill quality while achieving up to 1.16x end-to-end first-prefill speedup on an NVIDIA H100, corresponding to a 13.9% latency reduction.

1 Introduction

Recurrent-depth computation raises long-context prefill cost, while repeated quantization perturbs the evolving representation. RAMP allocates low precision early and restores precision for terminal refinement, following the finding that early errors are attenuated more by subsequent steps.

1 Introduction

  • Long-context prefill is costly because each recurrent step processes the full input sequence, and quantizing every step repeatedly perturbs the evolving representation.The paper asks how to allocate a fixed precision budget across recurrent depth while preserving full-precision fidelity.Source: §1 · pp. 1–2
  • At the same budget of 24 FP8 and 8 BF16 steps, RAMP’s terminal-BF16 schedule outperforms all 20 random schedules, while the reverse Early-BF16 schedule has substantially larger error.Figure 1 compares final hidden-state relative error across schedules.Source: Figure 1 · p. 2
  • Under locally contractive dynamics, earlier quantization errors undergo more subsequent attenuation, giving them a smaller effect on the final representation.The paper’s fixed-budget analysis therefore favors low precision early and high precision later; experiments support this prediction.Source: §1 · p. 2
  • RAMP is a training-free mixed-precision inference method that uses low-precision weight-and-activation computation for early refinement and restores original precision for terminal refinement.This placement limits quantized computation to early recurrent steps rather than applying it throughout the trajectory.Source: §1 · p. 2
  • Across contexts up to 4K, RAMP preserves near-full-precision prefill quality while achieving up to 1.16× end-to-end first-prefill speedup on an NVIDIA H100, a 13.9% latency reduction.The introduction reports this as the method’s overall result.Source: §1 · p. 2

2 Problem Statement

The problem statement defines recurrent-depth computation, the prefill stage and its latency measure, and weight-and-activation quantization as a lower-precision alternative for recurrent operations.

2.1 Recurrent-Depth Language Models

  • A recurrent-depth language model repeatedly applies a shared transformation to refine its internal representation; the final representation is processed by output layers to produce token logits.The same parameters are reused across recurrent steps, unlike a conventional depth-T Transformer with distinct parameters at each layer.Source: §2.1 · p. 3

2.2 Prefill in Recurrent-Depth Inference

  • First-prefill latency is the time to process an input sequence and produce logits for the first generated token.During prefill, the complete sequence is processed before autoregressive token generation begins, and its representation is processed at every recurrent step.Source: §2.2 · p. 3

2.3 Weight-and-Activation Quantization

  • Weight-and-activation quantization maps both activations and weights to lower-precision representations for computation.A precision schedule assigns a numerical precision to each recurrent step; uniform schedules use one low-precision format throughout, while mixed-precision schedules vary precision across depth.Source: §2.3 · p. 3

3 Quantization Error Analysis

Quantization errors are attenuated by subsequent recurrent steps, so their impact depends on where they enter the trajectory. Under a fixed quantization budget, placing low precision early and reference precision late minimizes the theoretical final-error bound.

3 Quantization Error Analysis

  • The analysis bounds error for arbitrary precision schedules and requires only local contraction within the operating region relevant to quantized inference.The quantized and reference trajectories are compared from a shared initial representation; the final-deviation statement assumes e0 = 0.Source: §3 · pp. 3–4, Then the recurrent deviation obeys · p. 4

Then the recurrent deviation obeys

  • Earlier quantization errors undergo more subsequent contraction, whereas errors introduced near the end retain a larger effect on the final representation.The error bound separates final deviation into contributions from individual quantized steps, attenuated by the recurrent steps that follow them.Source: Then the recurrent deviation obeys · p. 4
  • With exactly K quantized steps, placing quantized computation as early as possible minimizes the final-error bound, while placing it as late as possible maximizes it.Random or interleaved schedules fall between these extremes in the bound, and the low-to-high schedule has a tighter bound than full quantization.Source: Then the recurrent deviation obeys · pp. 4–5

4 RAMP

RAMP allocates precision across recurrent steps, using low precision early and restoring the original precision for terminal refinement. Its FP8–BF16 implementation leaves the recurrent architecture and depth unchanged and requires no retraining.

4 RAMP

  • RAMP uses low-precision computation for early recurrent refinement and restores the original precision for terminal refinement.This placement follows the fixed-budget result that early quantization minimizes the final-error upper bound.Source: §4 · p. 5

4.1 Depth-Adaptive Precision Schedule

  • The schedule switches from low precision to reference precision at step K, where K is less than the total recurrent depth T.The quantized recurrent operator is used before the switch, and the reference-precision operator is used afterward.Source: §4.1 · p. 6

4.2 FP8–BF16 Instantiation

  • The implementation uses 24 FP8 recurrent steps followed by 8 BF16 steps by default, with T = 32 and K = 24.Eligible dense linear operations use FP8 weights and dynamically quantized FP8 activations; hidden representations and surrounding operations remain BF16. RAMP requires no retraining and adds one precision transition.Source: §4.2 · p. 6

5 Related Work

Related work studies recurrent-depth capability, quantized inference, and mixed-precision allocation across network components. RAMP instead targets recurrent-depth prefill and allocates precision across repeated executions of the same shared block.

5.1 Recurrent-Depth and Looped Language Models

5.2 Quantized Large Language Model Inference

5.3 Mixed-Precision and Prefill-Efficient Inference

  • Prior mixed-precision methods allocate precision across layers, operators, channels, or tensors, whereas RAMP allocates it across recurrent time.RAMP targets prefill latency, which matters for time to first token; prefill processes many prompt tokens in parallel and is typically compute-intensive [Agrawal et al., 2024; Zhong et al., 2024; Patel et al., 2024].Source: §5.3 · p. 7

6 Experiments

Experiments show that mixed-precision placement, not just the precision budget, shapes fidelity: RAMP uses low precision early and BF16 for terminal refinement. It retains near-full-precision quality across evaluated datasets and long contexts while improving long-context prefill speed.

6.1 Experimental Setup

  • The evaluation uses Huginn-0125 at recurrent depth 32, paired nested-context cohorts, fidelity measures relative to BF16-32, and H100 first-prefill timing against the same optimized prefill path.The setup includes WikiText and PG19 fidelity cohorts, random fixed-budget precision schedules, and native FP8 weight-and-activation arithmetic for systems timing.Source: §6.1 · pp. 7–8

6.2 RAMP Preserves Full-Precision Prefill Quality Across Datasets

  • Across WikiText and PG19, RAMP stays closer to BF16-32 than uniform FP8 across language-modeling loss, representation error, logit error, and top-1 agreement.The matched-budget comparison places the same 8 BF16 steps at the beginning in Early-BF16 or at the end in RAMP; PG19's likelihood difference is not statistically resolved, although representation ordering remains pronounced.Source: §6.2 · pp. 8–9, Table 1 · p. 8
  • Across WikiText contexts from 128 to 4096 tokens, paired ∆NLL remains near zero and hidden-state and logit errors remain stable; PG19 shows stable representation and output errors over 512–4096 tokens.PG19 ∆NLL stays small with a mild positive trend toward longer contexts.Source: §6.2 · pp. 9–10

6.3 End-to-End Long-Context Prefill Efficiency

  • At 4K context, RAMP reduces first-prefill latency from 1103.444 ms with BF16 to 949.847 ms, a 1.1617× speedup and 13.9% latency reduction.Gains are also reported at 1K and 2K contexts.Source: Table 2 · p. 10, §6.3 · p. 10
  • RAMP becomes faster than BF16 between 256 and 512 tokens, with gains stable from 1K onward as low-precision overhead is amortized by larger recurrent workloads.At short contexts, RAMP is slower because low-precision execution overhead is large relative to the recurrent matrix computation being accelerated.Source: §6.3 · p. 10

6.4 Precision Placement Matters

  • With the precision budget fixed at 24 FP8 and 8 BF16 steps, RAMP achieves lower hidden-state error than random schedules or Early-BF16, with the same qualitative advantage for paired NLL and final-logit error.Later BF16 placement is associated with higher fidelity across random schedules, consistent with the local-contraction prediction that early errors have more subsequent steps for attenuation.Source: Table 3 · p. 10, §6.4 · p. 11

6.5 Precision-Switch Ablation

  • A terminal phase of 8 BF16 steps removes most of the uniform-FP8 representation error; extending that phase brings comparatively small additional recovery.The default K = 24 switch keeps three quarters of recurrent steps in FP8 while recovering most of the fidelity gap to full BF16.Source: §6.5 · p. 11, Table 4 · p. 11

7 Conclusion

The conclusion links an error-propagation analysis to RAMP, a training-free mixed-precision inference method. It reports near-full-precision prefill quality through 4K context alongside up to 1.16× H100 first-prefill speedup.

7 Conclusion

  • Under locally contractive recurrent dynamics, earlier perturbations receive more attenuation and yield a smaller final-error upper bound than perturbations near terminal depth.Perturbation experiments and fixed-budget precision-placement comparisons strongly support this depth asymmetry.Source: §7 · pp. 11–12
  • RAMP performs early recurrent refinement with FP8 weight-and-activation computation and restores BF16 for terminal refinement without training.The conclusion describes it as a training-free inference method.Source: §7 · p. 12
  • Across contexts up to 4K, RAMP preserves near-full-precision prefill quality while reaching up to 1.16× end-to-end first-prefill speedup on an NVIDIA H100.The reported speedup corresponds to a 13.9% latency reduction.Source: §7 · p. 12

A.1 Proof of Theorem 3.1

The proof handles quantized and reference-precision recurrent steps separately, then combines their error contributions. Starting from a shared initial representation, it repeatedly applies the joint recurrence to bound final error.

A.1 Proof of Theorem 3.1

  • For a quantized step, the proof bounds trajectory deviation by adding and subtracting the reference transformation and applying the triangle inequality.For a reference-precision step, both trajectories propagate through the same transformation.Source: §A.1 · p. 15
  • The proof combines the quantized-step and reference-step cases into a joint recurrence for the trajectory error.It then applies the recurrence repeatedly from the shared initial representation, where e0 = 0.Source: §A.1 · p. 15
  • Repeated substitution expresses final error as accumulated quantization-error terms weighted by subsequent recurrent factors, yielding Theorem 3.1.The proof collects the expanded terms before concluding the theorem.Source: §A.1 · p. 15

A.2 Proof of Theorem 3.2

The proof shows that, under a fixed quantization budget, choosing earlier recurrent steps minimizes the error bound, while choosing later steps maximizes it. Long-context results show RAMP remains close to BF16-32 across WikiText and PG19 evaluations.

A.2 Proof of Theorem 3.2

  • Because final-error weights are nondecreasing with recurrent depth, selecting the first K weights minimizes the bound and yields qRAMP.The proof considers schedules with exactly K quantized steps.Source: §A.2 · p. 16
  • Selecting the final K weights instead yields qRev and the largest possible sum; the bound is strictly worse than qRAMP’s when the exchanged weights differ.Strict ordering holds when relevant contraction factors lie between 0 and 1 and an exchanged pair spans a strictly contractive step.Source: §A.2 · p. 16

A.3 Full Long-Context Fidelity Results

  • Across 128–4096 tokens on WikiText and 512–4096 tokens on PG19, RAMP stays close to BF16-32 with stable hidden-state and logit errors.Paired ∆NLL remains near zero on WikiText and small on PG19, where it trends mildly upward at longer contexts.Source: §A.3 · p. 17, Table 5 · p. 17, Table 6 · p. 17

Related papers

Loading 2610.13041v1…