PaperlayerView exact revision on arXiv

Source-linked AI summary

DuoLM: Fast and Effective Personalized Recommendation with Language Models

Jiang Wu, Yutong Song, Hongbo Wang, Guanghe Cao, Jeff Lei Liang

arXiv:2610.24481v2cs.IR

TL;DR

DuoLM addresses the online computation and serving-throughput challenges of evaluating personalized recommendations from full-text inputs. It combines offline reusable item representations with a personalized ranker and an optimized runtime, achieving teacher-comparable recommendation quality with up to 60.1× the teacher's vLLM throughput on a single H100.

  • Problem

    Full-text personalized recommendation requires substantial online computation, while efficient ID-based methods do not directly incorporate textual semantics or pretrained world knowledge.

    Source: §1 · p. 1

  • Method

    DuoLM uses an encoder LLM to create reusable item vectors offline and a personalized ranker LLM to score candidates with user history and feedback, alongside runtime optimizations.

    Source: §1 · p. 2

  • Results

    On a single H100, DuoLM delivers up to 60.1× the throughput of the teacher served by vLLM while achieving recommendation quality comparable to the teacher.

    Source: §6 · p. 9

  • Takeaways & Limitations

    DuoLM combines reusable item representations with shared history computation to deliver recommendation quality comparable to the teacher and higher serving throughput.

    Source: §6 · p. 9

    Evaluation focuses on Amazon Books and Video Games with 0.6B backbones and fixed candidate sets containing randomly sampled unobserved items.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Language models can make personalized recommendations from rich item content, user history, and feedback. However, evaluating many candidates with full-text inputs incurs substantial online computation and limits serving throughput. We introduce DuoLM, a framework for fast and effective personalized recommendation that integrates a compact recommendation model with an efficient serving runtime. An encoder LLM learns compact item representations for offline reuse, while a personalized ranker LLM combines them with user history and feedback for recommendation. We train both models end to end with recommendation supervision and teacher distillation. The runtime further accelerates inference through virtual token IDs for embedding access, shared history computation, and CUDA Graph replay. Benchmark results show that our framework achieves recommendation quality comparable to that of the expert teacher model. On a single H100, it delivers approximately 26,400 candidate scores per second, about $60\times$ the throughput of the teacher served with vLLM and automatic prefix caching.

1 Introduction

DuoLM addresses the challenge of combining efficient online serving with high-quality personalized recommendations by learning reusable item representations and pairing them with runtime optimizations. It reports recommendation quality comparable to the expert teacher alongside substantially higher throughput.

1 Introduction

  • Efficient personalized recommendation must reduce online computation without sacrificing quality, while also avoiding repeated history preparation and submission across candidates.ID-based recommenders can omit textual semantics, while full-text cross-encoders use context-dependent candidate representations; separate candidate submissions may repeat history work and scheduling overhead.Source: §1 · p. 1
  • DuoLM jointly learns reusable item representations and personalization, then uses runtime optimizations to translate computational savings into faster serving.The encoder compresses each item's text into one vector; the ranker uses candidate and history-item vectors with textual feedback to produce personalized scores.Source: §1 · p. 2
  • The encoder and ranker are co-trained with recommendation labels and distillation from a frozen full-text teacher, after which item representations are precomputed offline.Cross-entropy encourages selection of the observed target, while KL divergence encourages matching the teacher's preference distribution over the same candidates.Source: §1 · p. 2
  • Virtual token IDs reference GPU-resident precomputed embeddings, letting requests carry item IDs and feedback instead of full vectors and reducing rating-benchmark payloads by over 90%.The runtime resolves the IDs into continuous ranker input embeddings.Source: §1 · p. 2
  • The multi-item scoring path prefills user history once and shares its prefix KV cache across candidate branches while preserving independent candidate scoring.A tree-structured attention mask lets each candidate attend only to shared history and its own branch.Source: §1 · p. 2
  • Segmented prefill CUDA Graph replay reduces CPU scheduling and kernel launch overhead.The graphs capture GPU operations and their dependencies for replay.Source: §1 · p. 2

2 Method

DuoLM pairs a compact item encoder with a personalized ranker, training them jointly with recommendation labels and teacher distillation. Its serving runtime reuses item representations and shares history computation while scoring candidates independently.

2.1 Model Architecture

  • The encoder produces user-independent item representations, which the ranker combines with ordered history embeddings and feedback to score candidates.A shared scoring head ranks candidates without autoregressive decoding.Source: §2.1 · pp. 2–4

2.2 Training

  • The encoder, interface, and ranker are jointly trained using target-selection cross-entropy and KL distillation from a frozen full-text teacher.The teacher’s cached scores supply relative preferences over each candidate group.Source: §2.2 · p. 4

2.3 Efficient Inference

  • Virtual token IDs retrieve precomputed item embeddings from a GPU-resident table, reducing average rating-benchmark request payloads by over 90% versus full-vector transmission.Online inputs carry item IDs and feedback rather than full item vectors.Source: §2.3 · pp. 4–5
  • Multi-item scoring prefills each request’s history once and shares its prefix KV cache across independently scored candidate branches.A tree-structured attention mask limits each candidate to the shared history and its own branch.Source: §2.3 · p. 5
  • MIS removes (Kq −1)Lq(Lq + 1)/2 repeated prefix pairs, while CUDA Graph replay reduces repeated per-kernel host dispatch.The pair counts exclude delimiters, padding, and kernel overheads.Source: §2.3 · p. 5

3 Experiment Settings

The experiments evaluate rating prediction and next-item recommendation on Amazon Reviews 2023 Books and Video Games data. Comparisons use 0.6B Qwen3 backbones and include the trained full-text teacher as a quality reference.

Data and evaluation

  • Rating prediction is evaluated on Books across history lengths K ∈ {1, 3, 5, 10}, using NLL, MAEmed, MAEexp, RMSE, and RPS.The setup uses 10,000 training users and 1,000 development users.Source: §3 · p. 5
  • Next-item recommendation is evaluated on Books and Video Games with HR@1, HR@10, and NDCG@10.The evaluation uses 5,000 Books users and 3,000 Video Games users.Source: §3 · p. 6

Model configuration

  • Methods use 0.6B Qwen3 backbones, with the trained full-text teacher as the quality reference; DuoLM’s default uses FIR with one vector per item.FIE encodes feedback with item content, whereas FIR supplies feedback as ranker text.Source: §3 · p. 6

A.1 Data Splits and Evaluation

The section specifies the datasets, evaluation protocols, model configurations, and serving measurements used in the experiments. On the serving trace, DuoLM and the full-text teacher achieve similar recommendation quality.

A.2 Model and Baseline Implementations

  • Recommendation uses LastToken with a scalar scoring head, while rating prediction uses Resampler/RMS and a five-star output head; item text is capped at 512 and 256 tokens, respectively.Both encoder and ranker use 28-layer, 1,024-dimensional Qwen3 backbones.Source: §A.2 · p. 12

A.3 Serving Measurements

  • Serving throughput includes online input preparation through postprocessing, but excludes offline encoding, startup, warmup, and candidate retrieval.Recommendation systems share an H100 and fixed request trace; rating comparisons use a different H100.Source: §A.3 · p. 13
  • On the serving trace, DuoLM reaches NDCG@10 of 0.5833 versus 0.5801 for the full-text teacher.Quality is measured using each user’s first response in the first run; the trace contains 1,000 fixed, scorable users.Source: §A.3 · p. 13

B.1 Within-User Ranking and History Length

DuoLM improves within-user ranking over the History prior, and its rating predictions improve with longer histories. The section also tests feedback use, input suffixes, and the training objective and settings.

B.1 Within-User Ranking and History Length

  • DuoLM improves within-user ranking over History prior on all three metrics, while the teacher has the highest NDCG@3 and pairwise agreement.This diagnostic ranks each user’s observed, rated target items by predicted expected rating.Source: §B.1 · p. 14, Table B1 · p. 14

B.2 Feedback Intervention

  • Shuffling or removing historical ratings increases NLL for the teacher, FIE, and FIR, with removal having the larger effect.Shuffling preserves item order and the rating multiset; removal preserves the history items.Source: §B.2 · p. 14, Table B3 · p. 14

B.3 Input Suffix Templates

  • The main text uses the short input suffix to reduce the number of input tokens.The suffix comparison changes only the input template while keeping FIR weights fixed.Source: §B.3 · pp. 14–15

Related papers

Loading 2610.24481v2…