PaperlayerView exact revision on arXiv

Source-linked AI summary

GQLA: Group-Query Latent Attention for Hardware-Adaptive Large Language Model Decoding

Fanxu Meng

arXiv:2605.15250v3cs.LGcs.AI

TL;DR

MLA’s single absorbed-MQA decoding path limits hardware adaptability, tensor parallelism, and MTP on commodity GPUs. GQLA exposes equivalent MQA and GQA paths from shared weights, with analytical predictions that each matches the H100 or H20 roofline while enabling zero-redundancy tensor parallelism.

  • Problem

    MLA’s absorbed-MQA-only design couples efficient decoding to H100-class hardware ratios, limits head-axis tensor parallelism, and provides no MTP gain on commodity GPUs.

  • Method

    GQLA indexes latent up-projections by groups, exposing equivalent MQA-absorb and per-group GQA decoding paths, while TransGQLA converts pretrained checkpoints without pretraining from scratch.

  • Results

    The analytical Roofline model predicts that shared GQLA weights reach both H100 and H20 ridges, while the GQA path enables 8-way zero-redundancy tensor parallelism.

  • Takeaways & Limitations

    GQLA provides one pretrained-weight design with hardware-selectable decoding paths for compact-latent inference, per-group traffic, and theoretical zero-redundancy tensor parallelism.

  • Takeaways & Limitations

    Hardware throughput, latency, tensor-parallel communication, and MTP speedups are modeled analytically rather than measured on devices or multi-GPU deployments.

Abstract

from arXiv · show

Multi-head Latent Attention (MLA), the attention used in DeepSeek-V2/V3, jointly compresses keys and values into a low-rank latent and matches the H100 roofline almost perfectly. Its trained weights, however, expose only one decoding path - an absorbed MQA form - which ties efficient inference to H100-class compute-bandwidth ratios, forfeits tensor parallelism along the head axis, and yields no Multi-Token Prediction (MTP) gain on commodity inference GPUs such as the export-restricted H20. We propose Group-Query Latent Attention (GQLA), a minimal modification of MLA whose trained weights expose two algebraically equivalent decoding paths over the same parameters: an MQA-absorb path identical to MLA's, and a GQA path with a per-group expanded cache. The runtime picks the path that matches the target hardware - no retraining, no custom kernels - so a single set of GQLA weights pins the rooflines of both H100 (MQA-absorb, s_q=1) and H20 (GQA + MTP, s_q=2), while supporting up to 8-way zero-redundancy tensor parallelism on the GQA path. To avoid pretraining from scratch we extend TransMLA into TransGQLA, which converts a pretrained GQA checkpoint into a GQLA model; on LLaMA-3-8B it compresses the per-token KV cache to 28.125% of the GQA baseline on the MQA-absorb path while structurally preserving GQA-level traffic on the per-group path.

1 INTRODUCTION

GQLA addresses MLA’s hardware coupling and tensor-parallelism limitations by exposing equivalent MQA-absorb and GQA decoding paths over shared weights. Roofline analysis predicts hardware-specific operating points, while TransGQLA enables training-free conversion from pretrained MLA and GQA checkpoints.

  • Motivation: MLA’s MQA-absorb decoding is hardware-coupled to H100-class compute-bandwidth ratios and funnels heads through one shared latent, limiting tensor parallelism.On H20, the ridge drops to ∼37 FLOPs/byte, rendering MLA compute-bound.
  • GQLA: GQLA indexes latent up-projections by g groups, enabling two equivalent decoding paths: MQA-absorb with a latent cache and GQA with per-group KV caches.Both paths reuse existing MLA/GQA kernels.
  • Roofline analysis: With hq =128, g=8, and one MTP head, analysis predicts H100 + MQA-absorb at sq =1 and H20 + GQA at sq =2 reach their respective ridges.The H20 configuration retains near-full MTP speedup; these are analytical predictions, not measured GPU throughput.
  • GQLA: The GQA path enables 8-way zero-redundancy tensor parallelism while preserving the shared GQLA weights across decoding modes.The two paths target different hardware operating points without requiring custom kernels.
  • TransGQLA: TransGQLA converts pretrained GQA and MLA checkpoints without gradient updates, using targeted head-merging changes for GQA and side-separated PCA for MLA.The method extends TransMLA and adds similarity-based head grouping and Hessian-weighted PCA.

2 RELATED WORK

GQLA builds on established KV-cache reduction methods and roofline-driven attention analysis, while extending latent attention with a restored GQA decoding path and a conversion pipeline for pretrained models.

  • KV-cache reduction via attention design: MQA uses one KV head, GQA shares one KV head per group, and MLA compresses keys and values into a low-rank latent with decoupled RoPE.GQLA remains in this architectural family by inheriting MLA’s latent compression while restoring MLA’s omitted GQA path.
  • Roofline-driven attention design: Roofline studies motivate hardware-aware latent-attention design, as inference becomes increasingly bandwidth-limited while compute scales faster than HBM.GQLA follows this methodology and extends it from H100 to H20 to motivate hardware-adaptive path selection.
  • Algebraically equivalent decoding paths: GQLA provides two algebraically equivalent decoding paths over one trained-weight set: GQA materializes g K/V groups, while MQA-absorb folds W_UK and W_UV into Q/O.The paths target H20 and H100 working points, respectively, and differ only by small BF16 finite-precision deviations of approximately 10^-3 per layer.
  • Converting pretrained MHA/GQA models: TransGQLA reuses TransMLA’s pretrained GQA-to-MLA conversion pipeline, modifying head merging so the GQA path and tensor parallelism survive.It also adds Hessian-weighted PCA as a route-agnostic calibration-only refinement applied on both conversion paths.

3 METHODS

GQLA uses shared latent parameters to expose equivalent GQA and MQA-absorb decoding paths, while TransGQLA converts pretrained GQA or MLA checkpoints without pretraining from scratch. Its conversion pipeline preserves group structure and applies RoRoPE, frequency-aware and Hessian-weighted compression refinements.

  • Architecture: GQLA compresses token embeddings into cKV_t, expands it into g grouped K/V representations, and decomposes queries into hq heads with decoupled-RoPE paths.The K/V projections produce g groups of per-head dimension dh, while query projections produce hq heads; positional terms use per-head qR_h and shared kR_h paths.
  • Two equivalent decoding paths: The same weights support equivalent GQA and MQA-absorb decoding: the former materialises g K/V groups, while the latter folds up-projections into Q/O and uses the latent cache.Deployment switches between paths through a one-shot cache compression or expansion.
  • TransGQLA conversion: TransGQLA converts pretrained GQA and MLA checkpoints into GQLA without pretraining from scratch by preserving group-indexed up-projections and recovering MLA factorisation through group-wise PCA.For GQA conversion, dropping replication keeps the merged module standard GQA, retains exact MQA-absorb reachability, and preserves group-axis tensor parallelism.
  • Conversion refinements: RoRoPE aligns each group’s RoPE basis to a common basis while preserving attention scores, enabling a single decoupled key representation.Per-head orthogonal rotations are absorbed into matching query projections, and the resulting RoPE repeats every dh latent dimensions.
  • Conversion refinements: FreqFold performs PCA within RoPE frequency bands, while norm balancing and joint compression reduce K-NoPE and V jointly to rkv without starving either side of reconstruction rank.FreqFold preserves RoPE pair structure and makes the later physical split lossless; norm balancing rescales K-NoPE and V before joint PCA, then restores scales in the up-projections.
  • Conversion refinements: Hessian-weighted PCA replaces token-uniform covariances in both compression stages with an end-to-end-loss proxy, using Tikhonov damping of 10^-2 tr ¯Σ(s)/dim(x(s)) and fp64 eigendecomposition.The refinement also applies to MLA-to-GQLA’s per-group activation PCA and composes orthogonally with its similarity permutation.

4 ROOFLINE ANALYSIS

Roofline analysis predicts that GQLA’s two decoding paths remain near peak across H100 and H20, unlike MLA, by matching MQA-absorb and GQA configurations to each hardware ridge. The recommended canonical configuration pairs H100 MQA-absorb with H20 GQA, while the predictions remain analytical rather than measured.

  • Roofline motivation: MHA has I ≈1 versus the H100 ridge I⋆≈295, while H20 lowers the ridge to ∼37 and makes H100-tuned intensity compute-bound.The Roofline throughput bound is min(I · BW, FLOPsmax), with I⋆ separating memory- and compute-bound regimes.
  • MQA-absorb path: MQA-absorb gives I ≈242 at sq =1 and 484 at sq =2; H100 step time rises from 2.82 to 4.61 µs, shrinking MTP speedup to ∼1.22×.Its cache traffic is independent of sq because all L tokens are read once and reused across queries.
  • GQA path: GQA reaches the H20 ridge at IGQA ≈38.8 for (g, sq)=(8, 2) and IGQA ≈37.6 for (g, sq)=(4, 1).IGQA scales linearly with sq and roughly inversely with g; the g=8 cache is 4224 bytes/token at the canonical configuration.
  • Hardware operating points: 221K tok/s is the predicted H20 throughput for both ridge-optimal GQA configurations, versus 65K tok/s for MLA, a 3.4× analytical improvement without retraining or custom kernels.All tok/s figures are Roofline predictions and have not been validated against on-device measurements.
  • Configuration tradeoffs: The recommended (g, sq)=(8, 2) configuration provides an 8-way TP cap, while (g, sq)=(4, 1) halves cache to 2176 bytes/token but permits only 4-way TP.Both remain deployable on H100 at 2.82 µs/step; combining g=4 with sq=2 MTP would require rkv ≤256.

5 EXPERIMENTS

Experiments evaluate calibration-only TransGQLA conversions across GQA and MLA backbones, finding strong cache compression and generally small capability losses, while showing that continued-pretraining recovery and real hardware performance remain unvalidated.

  • Conversion setup: 28.125% of the GQA baseline is the LLaMA-3-8B MQA-absorb cache after conversion, while the GQA-path cache remains comparable to the original.The converted model caches 576 elements per token on the MQA-absorb path; the per-group path uses 2gd_h + d_Rh.
  • MLA-route results: ∼3.3 Avg. pts is the zero-token commonsense loss for the full MLA→GQLA pipeline on GLM-4.7-Flash relative to its MLA teacher.The task-level changes are heterogeneous, including −7.4 MMLU and +0.2 OpenBookQA.
  • GQA-route results: Within 0.2% wikitext-2 PPL and ≤0.1 commonsense-average point, vanilla GQA→GQLA matches the TransMLA baseline on LLaMA-3-8B and Qwen2.5-7B.Hessian PCA then reduces LLaMA-3-8B PPL from 25.87 to 18.60 and Qwen2.5-7B PPL from 8.41 to 8.25.
  • MLA-route results: 68.28 is the commonsense Avg achieved by sim+Hess on GLM-4.7-Flash, outperforming the lower-PPL neigh+Hess configuration at 67.22.All four refinement cells remain within ±0.5 PPL of the MLA teacher’s 11.38; sim+Hess has PPL 10.95.
  • Robustness and limitations: ∼10−3 is the BF16 absorbed-vs-PCA gap in the GLM-4.7-Flash per-layer sanity check, and it does not compound across the ∼60 layers.Continued-pretraining recovery was not evaluated and remains future work.
  • Robustness and limitations: No measured hardware performance is reported: throughput, latency, and profiler-level HBM-traffic/FLOPs results are derived from an idealised Roofline model rather than real H100/H20 deployments.The model omits kernel-launch overhead, actual attention-kernel utilisation, batching, and KV paging effects.

6 CONCLUSION

GQLA addresses three hardware drawbacks of MLA’s MQA-absorb-only design by exposing compact-latent MQA-absorb and per-group expanded GQA decoding paths from shared trained weights. TransGQLA extends this design to pretrained checkpoints without gradient updates, using loss-weighted PCA and similarity grouping that improve GLM-4.7-Flash commonsense Avg by +1.06 to +1.72 points.

  • 6 CONCLUSION: GQLA fixes MLA’s hardware coupling, lost head-axis tensor parallelism, and zero MTP gain on commodity GPUs with two algebraically equivalent decoding paths.Its paths are a compact-latent MQA-absorb route identical to MLA’s and a per-group expanded GQA route, enabled by indexing up-projections by group rather than query head.
  • 6 CONCLUSION: TransGQLA enables conversion from pretrained checkpoints with no gradient updates on either decoding route.It makes the GQLA design accessible without requiring pretraining from scratch.
  • 6 CONCLUSION: +1.06 to +1.72 commonsense Avg pts on GLM-4.7-Flash come from Hessian-weighted PCA and similarity grouping.Hessian-weighted PCA replaces uniform-token activation PCA, while Procrustes-aligned similarity grouping replaces neighbor head merging when hq > g.
Loading 2605.15250v3…