PaperlayerView exact revision on arXiv

Source-linked AI summary

hLLM: Single Pass Decoding for Generative Reranking

Emil Laftchiev, Prachi Agrawal, Moe Kayali, Bixing Yan, Qi Xu, Zijie Lei, Chen Qiu, Zhi Hua, Ke Li, Luke Simon

arXiv:2609.01807v1cs.LGcs.AIcs.IR

TL;DR

Autoregressive generative rankers incur sequential decoding costs even though their outputs are structured permutations. HLLM reads item–position scores from prefill hidden states and applies Hungarian assignment, reaching 28 ms end-to-end with ranking quality on par with the teacher.

  • Problem

    Autoregressive ranking spends sequential forward passes to emit ordinal tokens, although the output is a fixed-length permutation whose candidates are known in advance.

  • Method

    HLLM uses a lightweight self-attention head to form an N × K item–position score matrix from prefill hidden states and decodes it with Hungarian assignment.

  • Results

    28 ms end-to-end inference delivers a 64× speed-up over the reasoning teacher while maintaining near-lossless ranking quality on proprietary and open-source datasets.

  • Takeaways & Limitations

    HLLM connects generative ranking with combinatorial optimization, providing constant-pass decoding and permutation validity by construction.

Abstract

from arXiv · show

Large language models (LLMs) achieve state-of-the-art generative ranking quality, but the ranking they produce must be decoded, and autoregressive decoding spends one sequential forward pass per emitted token. We observe that the only tokens a ranker must emit are the $N$ ordinal values naming the items in ranked order, and that this narrow, permutation-structured output format admits decoding strategies which are much more efficient than left-to-right generation. We introduce hLLM (Hungarian LLM), a format-specialized decoding strategy that decodes all $N$ ordinals in $O(1)$ forward passes. hLLM reads an $N \times K$ item-position score matrix off the LLM's prefill hidden states with a lightweight self-attention head, then decodes the ordinals as the optimal bipartite assignment of that matrix via the Hungarian algorithm, yielding a valid permutation by construction rather than by repair. Through a systematic study of training signals and backbone adaptation, we show that LoRA-based fine-tuning combined with teacher ranking distillation reaches 28 ms end-to-end inference, a speed-up of $64\times$ while maintaining ranking quality on par with the teacher. We provide a complete ablation decomposing the contributions of architecture, training signal, and backbone adaptation. Our framework connects generative ranking to combinatorial optimization, opening a path toward other $O(1)$-decode mechanisms for real-time ranking.

1 INTRODUCTION

Generative ranking must decode a permutation of candidate ordinals, but autoregressive decoding scales sequentially with the output. HLLM extracts ranking information from prefill states and uses optimal assignment for constant-pass, valid decoding.

  • Generative ranking jointly orders N candidates for recommendation, advertising, and search, emitting each item’s ordinal exactly once.
  • Autoregressive decoding uses O(N · T) sequential passes for N ordinals, creating a re-ranking latency bottleneck despite parallel prefill.
  • HLLM asks whether all N ordinals can be decoded in O(1) forward passes while recovering the ranking computed by an autoregressive LLM.
  • A lightweight self-attention head converts prefill hidden states into an N × K item–position score matrix, which the Hungarian algorithm decodes as an optimal assignment.
  • The assignment formulation guarantees a valid permutation by construction and removes the sequential decode dependency through constant-pass inference.
  • 64× speed-up over an autoregressive LLM is achieved on proprietary data while maintaining ranking performance, with similar speedups on Amazon Beauty.

2 BACKGROUND AND RELATED WORK

Prior generative ranking methods largely retain autoregressive or iterative decoding, while HLLM exploits permutation structure for exact constant-pass decoding. Its distinguishing choices are hidden-state assignment decoding and inference-time permutation enforcement.

  • LLM-based ranking and generative retrieval: RankGPT introduced generative reranking with autoregressive permutation decoding, while FIRST uses first-token logits for a closer constant-pass alternative.
  • Decoding strategies: Non-autoregressive generation can speed decoding but typically uses refinement passes and may degrade quality; HLLM instead solves a constrained permutation problem exactly.
  • Training for permutation prediction: HLLM distills full permutations through a Sinkhorn cross-entropy objective and identifies an interaction between signal richness and backbone capacity.
  • Hungarian assignment in deep learning: Unlike DETR, HLLM applies Hungarian matching at inference to hidden states as the decoder, enforcing the permutation constraint during decoding rather than restoring it afterward.
  • Efficient LLM inference: Speculative decoding, Medusa, EAGLE, and LayerSkip preserve left-to-right generation, so their pass counts still scale with output length.
  • Efficient LLM inference: No existing method is described as combining constant-pass ranking, real-time latency, near-lossless quality, and permutation guarantees by construction.

3 PROBLEM FORMULATION

HLLM formulates ranking as optimal assignment over a score matrix and uses differentiable training with exact Hungarian decoding at inference. This replaces sequential decoding with constant-pass ranking while preserving valid permutations.

  • 3.1 NOTATION AND SETUP: An autoregressive teacher generates a permutation through T sequential decode passes, whereas the student seeks the same ranking in O(1) decode passes.The task orders N candidates retrieved for a user context.
  • 3.1 NOTATION AND SETUP: 28 ms prefill remains fixed while autoregressive decoding scales with generated tokens, making decoding rather than candidate representation the latency bottleneck.With reasoning, T ≈657 tokens; without reasoning, T ≈39.
  • 3.2 DECODING A RANKING AS OPTIMAL ASSIGNMENT: The score matrix M assigns affinities between N items and K rank positions, and the optimal ranking is their maximum-weight bipartite matching.Items index one side of the graph and rank positions index the other.
  • 3.2 DECODING A RANKING AS OPTIMAL ASSIGNMENT: The linear assignment problem is solved exactly by the Hungarian algorithm in O(N^3), while LAPJV achieves O(N^2) average-case complexity.Both methods provide combinatorial optimization procedures for decoding the assignment.
  • 3.2 DECODING A RANKING AS OPTIMAL ASSIGNMENT: For N = 50, the teacher requires 657 sequential forward passes, while the Hungarian solver uses roughly 8 µs on CPU.The full HLLM pipeline uses one prefill plus one Hungarian solve.
  • 3.3 DIFFERENTIABLE RELAXATION VIA SINKHORN: Because Hungarian decoding is non-differentiable, training uses Sinkhorn to provide a continuous relaxation through doubly-stochastic matrices.Sinkhorn projects a non-negative matrix onto the Birkhoff polytope by alternating row and column normalization.
  • 3.3 DIFFERENTIABLE RELAXATION VIA SINKHORN: The Sinkhorn cross-entropy trains against a teacher permutation matrix, while inference applies Hungarian decoding directly to M for an exact permutation.As τ approaches zero, the relaxation recovers a permutation matrix.

4 METHOD: HLLM

HLLM processes candidate representations in one prefill pass, scores item-position compatibility, and uses Hungarian assignment to produce a valid ranking. Its method combines a self-attention scoring head, assignment decoding, teacher-derived targets, and LoRA adaptation.

  • 4 METHOD: HLLM: HLLM’s winning configuration uses a self-attention head with L = 2 and a LoRA-adapted backbone.Alternative head variants are evaluated separately.
  • 4 METHOD: HLLM: The LoRA-adapted decoder-only backbone processes the user context and all candidates in one forward pass and supplies one hidden state per candidate.The model has 0.6B parameters and gathers each candidate’s final description token.
  • 4.1 ARCHITECTURE OVERVIEW: A single prefill produces per-item hidden states that the scoring head maps into an N × K item-position score matrix.The matrix is then decoded by the Hungarian algorithm in one shot.
  • 4.1 ARCHITECTURE OVERVIEW: The scoring-head variants differ in whether items are scored independently, compared explicitly, or attended by learnable position embeddings.The self-attention head performs explicit item-to-item comparison.
  • 4.1 ARCHITECTURE OVERVIEW: Hungarian decoding turns the model’s score matrix into an optimal one-to-one assignment and therefore a valid permutation in one shot.Unlike DETR, HLLM uses Hungarian matching as the inference-time decoder rather than for training loss computation.
  • 4.2 SCORING HEAD VARIANTS: The study evaluates three scoring-head architectures on identical frozen or LoRA-adapted backbone hidden states.This isolates the contribution of cross-item comparison.
  • 4 METHOD: HLLM: Training first stores greedy teacher permutations offline, then trains the student against those fixed labels without online teacher-student interaction.The train-time and inference-time input distributions are identical by construction.
  • 4 METHOD: HLLM: LoRA rank-64 adaptation updates backbone projections so hidden states can encode ranking-relevant information while preserving pretrained knowledge.The adaptation is described as enabling the backbone to absorb richer teacher signals.

5 EXPERIMENTS

Experiments show that HLLM preserves ranking quality while substantially reducing latency, and that performance depends on combining rich teacher signals with an adaptable backbone and self-attention scoring head.

  • Main results: HLLM reaches 28 ms inference, 64× faster than the reasoning teacher and 3.1× faster than the no-reasoning teacher.The same ordinal output is decoded in one pass, isolating the decoding-strategy benefit in the no-reasoning comparison.
  • Main results: HLLM is indistinguishable from both teachers on NDCG@1 and Recall@1, while retaining 99.9% of teacher quality on AUC.NDCG@1 is 0.1791 versus 0.1776 / 0.1779, and Recall@1 is 0.1652 versus 0.1634 / 0.1635.
  • Signal and backbone ablation: Freezing the backbone or training only on click labels causes a significant ranking-performance penalty; their combination with rich permutation supervision succeeds.The ablation identifies an interaction between signal richness and backbone adaptability.
  • Head architecture ablation: The self-attention scoring head outperforms a linear probe and a head with learned position-dependent embeddings.This result holds on the LoRA-adapted backbone.
  • Latency decomposition: The LAPJV solver contributes 0.008 ms, under 0.03% of total latency, while the scoring head adds under 0.1 ms.At slate sizes N ≤50, the solver runs in microseconds and is effectively overlapped by the GPU forward pass.

6 ANALYSIS

HLLM’s training and decoding design jointly explain its ranking behavior: adaptable representations support rich permutation supervision, while constrained assignment guarantees valid deterministic outputs.

  • Representation and training: LoRA closes the capacity gap needed for hidden states to encode fine-grained pairwise preferences required by full permutation targets.The adaptation uses roughly 4M trainable parameters on a 600M backbone.
  • Structural guarantees: The Hungarian decoder guarantees 100% valid permutations, prevents duplicate position assignments, and makes inference deterministic.These properties are enforced by the one-to-one assignment constraint rather than learned or repaired afterward.

7 CONCLUSION

The paper presents HLLM as a constant-pass decoder that extracts complete rankings from hidden states, achieving near-lossless quality with large inference speedups across two datasets.

  • Conclusion: HLLM decodes all N item ordinals in O(1) forward passes and reaches 28 ms end to end.The method uses Hungarian decoding directly from hidden states.
  • Conclusion: Across a proprietary dataset and an open-source benchmark, HLLM achieves near-lossless ranking quality with 45–64× inference speedups.The study also profiles architecture components and separates backbone from self-attention contributions through ablations.
Loading 2609.01807v1…