PaperlayerView exact revision on arXiv

Source-linked AI summary

CRIM: Composable and Reusable Item Memory for Efficient Personalized Retrieval

Haolin Li, Haipeng Zhang, Mang Li, Zhaohui Xia, Xun Zhu, Miao Li, Yu Zhang, Ji Wu

arXiv:2610.35323v1cs.IRcs.CL

TL;DR

Growing interaction histories and repeated item-text encoding raise personalized retrieval costs, while compression must preserve semantics and interaction patterns. CRIM separates reusable item memories from request modeling, adds sequence supervision and sampled live re-encoding, and achieves strong retrieval with lower inference costs across its evaluated datasets.

  • Problem

    Growing histories and repeated item-text encoding increase personalized retrieval costs, while compressed representations must preserve item semantics and interaction patterns.

    Source: §1 · pp. 1–2

  • Method

    CRIM encodes items into reusable continuous memories, composes them in history order with an optional query, and combines cached memories with sampled live re-encoding for joint training.

    Source: §1 · p. 2

  • Results

    Across LazSeq, Qilin, and MovieLens-1M, CRIM achieves strong retrieval quality; on LazSeq it reduces serving input length by 80.75% and delivers a 4.05× latency speedup over the same-scale dense reference.

    Source: §5 · p. 9

  • Takeaways & Limitations

    CRIM combines strong candidate recall and ranking quality with lower serving costs on LazSeq at the same backbone scale.

    Source: §1 · p. 2, §5 · p. 9

    The cost analysis assumes a fixed catalog, and training-time cache refreshes and later updates add to offline cost.

    Source: §4.4 · p. 8

Abstract

from arXiv · show

Large language models incur substantial training and inference costs in personalized search and recommendation, as growing interaction histories lengthen request inputs and repeatedly expose the same item text to the model. We propose \textbf{C}omposable and \textbf{R}eusable \textbf{I}tem \textbf{M}emory (CRIM), a framework that separates item-level semantic encoding from request-level history modeling. A small set of learnable summary tokens compresses each item into compact continuous memories, while segment-level readout tokens (REC tokens) introduce predictive supervision within the interaction history. These memories are independent of users and queries and can therefore be reused across requests. At inference time, CRIM composes cached memories in interaction order and incorporates an optional query to produce a personalized retrieval representation, reducing repeated item-text encoding while retaining request-dependent contextual modeling. We further introduce a joint training mechanism that combines cached memories with sampled live re-encoding. Re-encoding only a few history items per request allows the retrieval objective to jointly optimize the item and request encoders at limited computational cost, while periodic cache refreshes maintain consistency. Extensive evaluation and cost analysis on MovieLens-1M, Qilin, and our new LazSeq dataset show that CRIM achieves state-of-the-art retrieval performance with substantially lower inference costs at the same backbone scale, while limiting item re-encoding during joint training. We will publicly release our code and dataset.

1 Introduction

Long interaction histories and repeated item-text encoding increase personalized retrieval costs, while compressed representations must preserve item semantics and interaction patterns. CRIM separates reusable item encoding from request-level modeling, adds predictive sequence supervision and sampled live re-encoding, and reports strong retrieval with lower inference costs.

1 Introduction

  • Personalized retrieval must reuse item semantics while retaining request-dependent context, preserve semantics and interaction patterns in compressed representations, and jointly optimize item and retrieval encoders.Growing histories increase training and inference costs, and repeatedly encoding the same items adds overhead.Source: §1 · pp. 1–2
  • CRIM independently encodes items into compact memories reusable across users and requests, then composes them in interaction order with an optional query for personalized retrieval.The design moves repeated item-text encoding offline while preserving request-context modeling and pretrained item-text semantics.Source: §1 · p. 2
  • Summary tokens compress item semantics, while REC tokens predict subsequent interactions from history prefixes to provide supervision throughout the sequence.Sampled live re-encoding lets retrieval gradients update the item encoder through a few positions; cached positions train the request encoder, and periodic refreshes maintain cache alignment.Source: §1 · p. 2
  • CRIM achieves state-of-the-art retrieval and, at the same backbone scale on LazSeq, combines strong retrieval quality with lower latency and fewer serving tokens.The evaluation covers MovieLens-1M, Qilin, and LazSeq, and the cost analysis relates offline investment and request computation to memory reuse volume.Source: §1 · p. 2
  • The paper introduces LazSeq, a long-history retrieval dataset, and evaluates retrieval quality and computational efficiency across LazSeq, MovieLens-1M, and Qilin.The contributions include a cost analysis of memory reuse.Source: §1 · p. 2

2 Related Work

Prior work studies language-model-based retrieval, input compression, and caching, but CRIM focuses on reusable item-level memories composed into request-specific histories. Its training combines cached representations with partial differentiable re-encoding and periodic refreshes.

2 Related Work

  • Language-model retrieval methods model item text and interaction sequences, semantic identifiers, or coordinated search and recommendation tasks; CRIM instead targets repeated item encoding in long histories.Recformer jointly models item text and sequences; TIGER predicts semantic identifiers; GEMS [38] coordinates task-specific signals.Source: §2 · p. 2
  • Compression methods range from prompt and context compression to memory slots and soft tokens, whereas CRIM reuses individual item memories and adds prediction objectives within histories.The passage also contrasts Dual-purpose Semantic IDs [20], which reduce representation storage and transfer costs.Source: §2 · p. 2
  • Caching methods address gradient paths and representation freshness; CRIM re-encodes sampled history positions for joint encoder optimization and periodically refreshes cached memories.Graph Segment Training reuses historical embeddings, while REST separates forward refreshes from parameter updates.Source: §2 · pp. 2–3
  • CRIM’s item memory encoder produces reusable memories, and its request encoder composes them in interaction order with an optional query for retrieval against separately encoded candidates.This overview distinguishes item encoding from request-level composition.Source: Figure 2 · p. 3

3 Method

CRIM separates reusable item semantics from request-specific history modeling, then composes item memories in order with optional query context. It combines semantic compression, sequence-level supervision, and sampled live re-encoding to support personalized retrieval with lower repeated computation.

3.1 Task Definition and Framework

  • CRIM encodes item identity and text into fixed-size memories, caches them independently of users and queries, and composes them in history order with optional query context.Candidate representations are cached separately for full-catalog retrieval.Source: §3.1 · p. 3

3.2 Item Memory Encoding and Semantic Pretraining

  • Shared learnable summary tokens compress item identity and text, while restricted-attention reconstruction and full-context distillation train semantic memories with a frozen backbone.Semantic pretraining updates summary-token and item-identity parameters without requiring user interaction sequences.Source: §3.2 · pp. 3–4, Figure 3 · p. 4

3.3 Sequence-Aware Memory Composition

  • REC tokens follow contiguous history segments and predict the first item in the next segment, adding prefix-level supervision while history memories retain causal interactions.Query tokens attend to history memories, and the final EOS representation produces the request vector.Source: §3.3 · p. 4

3.4 Joint Training with Cached and Sampled Memories

  • Joint training samples up to κ history positions for live IME re-encoding and uses cached memories at other positions, so retrieval gradients reach the item encoder without re-encoding every item.At epoch end, the catalog is re-encoded to refresh item-memory and candidate caches.Source: §3.4 · pp. 4–5, Figure 4 · p. 5

3.5 Computational Savings and Cost Amortization

  • For long histories, CRIM’s request-encoding cost scales approximately with γ=(K+R/b)/L, with attention costs scaling approximately as γ²; net savings depend on candidate matching and input adaptation.After Q reaches the break-even volume Q*, inference savings cover the additional offline investment, and each further request saves Δc.Source: §3.5 · pp. 5–6

4 Experiments

Across LazSeq, Qilin, and MovieLens-1M, CRIM delivers strong retrieval results, while reducing online latency and cumulative compute. Ablations show benefits from sequence supervision, semantic pretraining, and cache-based joint optimization, with four memory vectors offering the best reported capacity trade-off.

4.1 Experimental Setup

  • Experiments compare ID-based, generative, and dense methods across query-conditioned LazSeq and Qilin retrieval and query-free ML1M recommendation, using full-catalog ranking metrics.Histories are capped at 500 for LazSeq and 100 for Qilin and ML1M; the study uses common request-level ML1M data protocols.Source: §4.1 · p. 6

4.2 The LazSeq Dataset

  • LazSeq provides approximately 30,000 personalized-search requests, histories up to 500 interactions, and a shared catalog of 49,993 distinct items.Fixed request splits and a title catalog support reproducible comparisons.Source: §4.2 · p. 6

4.3 Retrieval Results and Comparative Analysis

  • On LazSeq, CRIM leads Recall@50, @100, and @1000 at 0.5835, 0.6969, and 0.9330, respectively, but ranks second on NDCG@50 and @100.It improves over the strongest baseline at Recall@100 by 2.43%; Table 1 covers full-catalog retrieval for histories up to 500 items.Source: §4.3 · p. 6, Table 1 · p. 7

4.4 Efficiency and Cost Analysis

  • CRIM cuts mean latency from 367.45 to 90.76 ms (4.05×) and cumulative compute at one million requests from 146.33 to 71.63 GPU-hours (51.05% saving).The cumulative-compute scenario holds the deployed model and catalog fixed; refreshes and later updates add offline cost.Source: §4.4 · pp. 7–8

4.3 Retrieval Results and Comparative Analysis

  • On Qilin, CRIM leads NDCG@10, NDCG@100, MRR@100, and full-catalog MRR; relative to Qwen3-Embedding-4B, NDCG@100 rises from 0.4518 to 0.4772.Table 2 reports full-catalog retrieval for histories up to 100 items.Source: §4.3 · p. 6, Table 2 · p. 8

4.5 Ablation Studies

  • Segment supervision improves Recall@100 by 3.70 percentage points over its removal, while semantic pretraining improves it by 1.03 points over no pretraining.Across three seeds, segment supervision improves Recall@100 by 2.53–3.97 percentage points.Source: §4.5 · p. 8, Table 4 · p. 8
  • Re-encoding budgets κ=1, 4, and 8 yield Recall@100 between 69.56% and 69.69%; four memory vectors give the highest Recall@100 and NDCG@100 with a 2.560 GB cache.CRIM exceeds fixed-cache training by 3.07 percentage points in Recall@100; the cache is smaller than at K=16 or K=32.Source: §4.5 · p. 8

4.3 Retrieval Results and Comparative Analysis

  • On ML1M, CRIM leads all six reported metrics, with Recall@10 of 0.1183 versus 0.0719 for Ministral-8B-Instruct and Recall@100 33.61% above Qwen3-Embedding-4B.Table 3 reports full-catalog retrieval for histories up to 100 items.Source: §4.3 · pp. 6–7, Table 3 · p. 8

5 Conclusion

CRIM separates reusable item semantics from request-dependent history modeling. It combines compact item memories and in-history predictive supervision with sampled live re-encoding, achieving strong retrieval quality and lower serving costs.

5 Conclusion

  • CRIM separates reusable item semantics from request-dependent history modeling across LazSeq, Qilin, and ML1M.It achieves strong candidate recall and ranking quality across these datasets.Source: §5 · p. 9
  • Summary tokens compress items into continuous memories, while REC tokens provide predictive supervision within ordered histories.Cached memories and sampled live re-encoding jointly train the item-memory and request encoders with a small differentiable item-encoding budget.Source: §5 · p. 9
  • On LazSeq, CRIM reduces serving input length by 80.75% and delivers a 4.05× latency speedup over the same-scale dense reference.Source: §5 · p. 9

A Implementation and Evaluation Details

LazSeq construction filters requests using interaction-history length and query–history lexical overlap, then builds a shared catalog from retained item occurrences. Requests are split in processed stream order.

A.1 LazSeq Construction

  • Requests need at least 100 original interactions and query–history lexical overlap of at least 0.5, with the most recent 500 history items retained.Overlap is the fraction of retained query tokens appearing in concatenated recent history titles; the query is lowercased and split on whitespace, with one-character tokens discarded.Source: §A.1 · p. 10
  • Item frequencies in target and retained history positions determine the retained catalog, and requests remain only when their target belongs to it.The final shared catalog is the union of retained item IDs across the output splits.Source: §A.1 · p. 10
  • Training, validation, and test splits are filled in processed stream order at an approximate 8 : 1 : 1 ratio.The protocol provides a common candidate space across splits.Source: §A.1 · p. 10

A.2 Training and Evaluation Protocols

The appendix documents dataset and checkpoint protocols, then calibrates serving and amortization estimates using A100 measurements. Cost estimates include optimization, cache construction, inference, and service-volume conversions.

A.2 Training and Evaluation Protocols

  • LazSeq semantic pretraining runs for five epochs, with 32 items per GPU, two prefix tokens, and up to 1,024 catalog negatives per segment.The cache-refresh variants refresh candidate vectors every epoch.Source: §A.2 · p. 10
  • The queryless ML1M setup uses the mean hidden state of the final REC block and adds target-title generation loss with catalog-sampled retrieval negatives.It uses fixed pooled candidate encodings.Source: §A.2 · p. 10
  • Qilin has 7,144/893/893 training/validation/test requests and 159,477 catalog items, while ML1M has 3,437/429/431 requests and 3,566 items.LazSeq is split at the request level, so a user can occur in multiple splits; catalog membership is determined before the request split.Source: §A.2 · p. 10
  • Checkpoint selection uses validation Recall@3000 for LazSeq and Qilin and Recall@10 for ML1M, with selected epochs 3, 2, and 7, respectively.LazSeq ablations select checkpoints on validation Recall@3000 and evaluate the selected checkpoint once on test; full MRR is unavailable for beam-limited TIGER and GenSAR runs.Source: §A.2 · p. 10

A.3 A100 Measurements and Cost Calibration

  • The serving-length measurement counts RE input positions, including memory slots, REC tokens, query, and special tokens, after five warm-ups and two passes over 200 fixed requests.The cache is built before timing; retrieval is over 49,993 candidates.Source: §A.3 · p. 10
  • Figure 5 estimates 45.66 versus 43.96 GPU-hours for CRIM and the dense reference in optimizer costs, with initial cache construction taking 2,809.16 versus 206.64 seconds.Measured online costs are 90.66 versus 368.32 ms per request; these estimates cover optimizer computation, initial cache construction, and inference with a fixed model and catalog.Source: §A.3 · p. 10
  • For Figure 5’s service-volume inset, monthly and annual totals are divided by 30 and 365, respectively.Starred per-second rates are converted to daily equivalents by multiplying by 86,400.Source: §A.3 · p. 10

A.4 Additional Comparisons

The additional comparisons report a Qilin recall trade-off against Qwen3-Embedding-4B and GTE-1.5B, alongside a highest-Recall@100 result among ID-based models. The appendix also points to LazSeq construction and main-result training configurations.

A.4 Additional Comparisons

  • Among ID-based models, S3Rec has the highest Recall@100 at 0.1097.Source: §A.4 · p. 10
  • Figure 6 depicts LazSeq request filtering by query–history overlap and candidate retention based on item frequencies in history and target positions.Source: Figure 6 · p. 11
  • Table 5 presents the training configurations used for CRIM’s main results.Source: Table 5 · p. 11
  • On Qilin, Recall@10 is 0.5845 for CRIM versus 0.5801 for Qwen3-Embedding-4B, while GTE-1.5B achieves 0.5969.CRIM’s Recall@100 is 0.7458 versus 0.7592 for Qwen3-Embedding-4B; the passage characterizes CRIM’s Qilin benefit as better target ranking.Source: §A.4 · p. 11

B.1 Consistency across Random Seeds

Across seeds, segment supervision improves all three retrieval metrics. In re-encoding and cache-refresh experiments, CRIM leads all three metrics, with Recall@100 varying only slightly across tested re-encoding counts.

B.1 Consistency across Random Seeds

  • Mean Recall@100 is 69.68% for CRIM, 66.29% without segment supervision, and 66.43% with retrieval loss alone.Segment supervision improves all three metrics in every seed.Source: §B.1 · p. 11, Table 6 · p. 11
  • Omitting semantic reconstruction lowers mean Recall@100 by 0.56 points, NDCG@100 by 0.41 points, and MRR@100 by 0.26 points.The main retrieval benefit of the auxiliary objectives comes from segment supervision.Source: §B.1 · p. 11, Table 6 · p. 11

B.2 Live Re-encoding and Cache Refresh

  • Recall@100 is 69.56%, 69.66%, and 69.69% for κ=1, 4, and 8, respectively, and CRIM leads all three metrics.The table varies training-time re-encoding and item-memory refresh.Source: §B.2 · p. 11, Table 7 · p. 12

B.3 Memory Capacity and Storage

Four summary tokens are supported by the reported capacity trade-off, while sampled live re-encoding avoids the all-history memory failure in a controlled probe. Intervention and subgroup results indicate how retrieval varies with history composition and length.

B.3 Memory Capacity and Storage

  • Increasing K from 4 to 16 multiplies serving storage by 3.4 and mean input length by 3.37 without improving quality.At K=32, storage reaches 16.894 GB; the results support using four summary tokens per item.Source: Table 8 · p. 12, §B.3 · p. 12

B.4 Live Item Encoding

  • With κ=4, three optimizer steps average 68.45 seconds and use at most 23.35 GB per GPU, while all-history live encoding runs out of memory during warm-up.The controlled probe fixes batch, RE history, and loss configuration.Source: §B.4 · p. 12

B.5 History and Query Interventions

  • Within-history permutations reduce NDCG@100 by 0.35–0.52 points and MRR@100 by 0.36–0.51 points, with paired intervals excluding zero.The interventions retain the same item multiset; cross-user history swap and norm-matched mean memories also show Recall@100 advantages for original histories and item-specific memories.Source: Table 9 · p. 12, §B.5 · p. 12

B.6 History-Length and Item-Frequency Groups

  • CRIM has higher Recall@100 through 250-item histories, while the dense reference leads for 251–500 items; the aggregate difference is 1.70 points with a 95% interval of [−0.60, 4.03].Only seven requests fall in the training-frequency 1–10 group.Source: Table 10 · p. 12, §B.6 · p. 12

Related papers

Loading 2610.35323v1…