PaperlayerView exact revision on arXiv

Source-linked AI summary

RetrievalFormer: A Dual-Encoder Transformer for Efficient Approximate Nearest Neighbor Retrieval and Cold-Item Recommendation

Theodore Rogers, Joe Standerfer, Dmitrii Timoshenko, Haoxue Li, Zuhaib Akhtar, Soyoung Yang

arXiv:2608.24079v1cs.IRcs.LG

TL;DR

The paper asks how a shared search-and-recommendation index can score unseen items while limiting recommendation-quality and scaling costs. RetrievalFormer uses jointly trained transformer user and feature-based item towers with dot-product retrieval, and evaluates the trade-offs against retrained baselines. It preserves cold-item openness and performs strongly in strict cold-start tests, but exact-quality training remains constrained by catalog-scale memory.

  • Problem

    Shared search-and-recommendation indexes need to score new items from features because search has no exploration slot, while the recommendation cost of this openness lacks controlled quantification.

  • Method

    RetrievalFormer jointly trains a transformer user tower and feature-based item tower whose pre-computed embeddings support dot-product nearest-neighbor retrieval and zero-shot scoring of new items.

  • Results

    Across benchmarks, RetrievalFormer trades some warm accuracy for index openness, reaches 1.4× the strongest retrained dedicated cold-start method, and exact full-softmax improves Recall@20 over sampled InfoNCE.

  • Takeaways & Limitations

    Dual-encoder retrieval keeps the shared index open to new items and has no retrieval-stage serving-cost regression against ID-softmax retrieval.

  • Takeaways & Limitations

    Exact-quality training remains infeasible at the catalog scales a shared index demands, and the paper lacks head-to-head accuracy comparisons at those scales.

Abstract

from arXiv · show

A shared search-and-recommendation index must score new items from features alone because search has no exploration slot. In a public log covering both surfaces over one catalog, $38.6\%$ of held-out query-search impressions show an item never previously shown or visited. For user-cold engagements, the feature-based tower serves this demand without measurable loss against $99$ sampled negatives ($0.9595$ Recall@20 versus $0.9510$ warm). A lexical baseline reaches similar parity, while a full-catalog check remains statistically undecided. Dual-encoder retrieval therefore keeps the index \emph{open} to new items, unlike an ID-softmax recommender that requires retraining. We price this openness on recommendation against six sequential baselines, each retrained and tuned through five rounds on corrected targets. A float32 timestamp bug had reordered leave-one-out targets for $19.7\%$ of users. On MovieLens-1M, warm accuracy trails the strongest retrained baseline by $5.2\%$ Recall@20 and $11.4\%$ NDCG@20. On MIND, the gap narrows to $0.8$--$3.6\%$ relative to the five strongest baselines, though the model ranks sixth of seven. Under strict zero-leakage cold-start evaluation, the content tower achieves $0.172 \pm 0.006$ Recall@20, $1.4\times$ the strongest retrained dedicated method ($0.124 \pm 0.007$) and $3\times$ a training-free floor, without cold-specific training. Exact full-softmax training raises Recall@20 by $54\%$ on MIND-small and $6.9\%$ on MovieLens-1M over sampled InfoNCE, but recomputes the full catalog each step and exhausts accelerator memory at $240$K items. Approximate nearest-neighbor search explains none of the remaining gap, serving cost does not regress against ID-softmax retrieval, and a history-window sweep explains half the post-recipe remainder. Exact-quality training at catalog scale remains the open problem.

1 Introduction

RetrievalFormer adapts dual-encoder retrieval to shared search-and-recommendation indexes, preserving zero-shot scoring for new items while quantifying recommendation accuracy, scale, and evaluation trade-offs.

  • Motivation: 38.6% of held-out query-search impressions contain an item never previously shown or visited, motivating feature-based cold-item scoring on shared indexes.Search cannot use exploration slots to bootstrap unseen items, so cold items must be scored from content at the first request.
  • Motivation: ID-softmax recommenders have closed vocabularies: unseen items cannot be scored from features and catalog growth requires retraining.Their catalog-sized output scan is also linear in catalog size, but dual-encoder retrieval has the same serving-stage matmul cost.
  • Approach: RetrievalFormer uses a transformer user tower and feature-based item tower trained jointly for dot-product retrieval, with pre-computed item embeddings and zero-shot scoring of new items.The shared architecture supports both query-initiated and history-initiated retrieval through one item index.
  • Results: 94.8% of the strongest retrained MovieLens-1M baseline is achieved at 0.3739 ± 0.0013 Recall@20, with half the remaining gap attributed to history-window choice.The comparison uses six baselines retrained on corrected targets and tuned through five documented rounds.
  • Results: 1.4× the strongest retrained dedicated method is reached on cold Recall@20 under zero-leakage evaluation, while a simple frozen score fusion exceeds all three dedicated methods on cold and warm axes.The feature-based tower also sits 3× above a training-free content-similarity floor.
  • Evaluation: 19.7% of MovieLens-1M users had leave-one-out targets silently reordered by float32 timestamp quantization, and corrected retraining lowered every baseline score by 0.016–0.039.This makes cross-model comparisons across pipelines non-comparable unless the target construction is corrected.

2 Related Work

Prior work establishes strong transformer sequence modeling, retrieval architectures, joint search-recommendation systems, content-based cold-start methods, and ANN infrastructure; RetrievalFormer positions itself at their intersection with controlled evaluation.

  • Sequential recommendation: SASRec and BERT4Rec use self-attention over behavior sequences but predict through softmax over a closed item vocabulary.Items enter the output vocabulary only through training, preventing direct scoring of unseen item IDs.
  • Retrieval: Two-stage retrieval commonly uses lightweight matrix-factorization or two-tower models before precise ranking, while PinnerFormer provides a deployed transformer retrieval relative.Prior deployment reporting does not substitute for the controlled offline comparison supplied here.
  • Unified systems: Unified search-and-recommendation research shares user–item interactions across surfaces, models behavior-stream transitions, or autoregressively decodes item identifiers.These approaches differ in whether they preserve an indexable feature-based route to unseen items.
  • Cold start: Content-based cold-start methods use side information or dedicated training, but ID-softmax attribute models still cannot score unseen item IDs.UniSRec shares the premise that content encoders unlock unseen items, whereas RetrievalFormer evaluates the dual-encoder serving form under controlled protocols.
  • Evaluation: Reported gains often shrink against properly trained baselines, motivating one protocol, retraining, tuning, and a seen-item echo diagnostic.The diagnostic quantifies how evaluation-time masking choices move model scores.
  • ANN search: IVF, HNSW, and PQ enable large-scale vector search, while RetrievalFormer measures IVF-PQ fidelity against exact scoring rather than proposing a new index.The comparison isolates retrieval approximation from representation and training effects.

3 Methodology

RetrievalFormer encodes heterogeneous item features and user histories into a shared embedding space, then trains that space with sampled or full-catalog objectives for indexed retrieval.

  • Item tower: The item tower encodes heterogeneous features into a dense embedding that can be pre-computed and indexed.New items receive embeddings from their features without retraining.
  • User tower: The transformer user tower processes interaction sequences into a query embedding, and recommendations maximize the dot product between user and item vectors.The shared output dimension d is distinct from transformer width d_model and per-feature dimensions d_f.
  • Feature fusion: AttentionFusion applies multi-head self-attention with residual connections, layer normalization, and pooling over heterogeneous feature sets.The block is permutation-invariant, handles variable-length inputs, and fuses item metadata, interaction context, and user profiles.
  • Feature fusion: Shared embedding tables reuse categorical vectors across item metadata, history events, and user attributes, reducing embedding parameters by roughly 3× in the implementation.This also keeps feature semantics consistent across towers and supports cold-start generalization.
  • User tower: Each historical interaction becomes an enriched token combining item features with rating and temporal context before transformer processing.The best configuration uses prefix-LM attention, a last-token readout, and an item-ID pathway that is absent for cold items.
  • Training objectives: Sampled InfoNCE uses in-batch items plus k uniformly sampled catalog items as negatives, with per-step cost independent of catalog size.The objective balances positive-pair alignment and embedding-distribution uniformity, supporting ANN indexing.
  • Training objectives: Full-softmax cross-entropy scores all catalog items each step and directly optimizes full ranking, but its memory scales with catalog size because item outputs and gradients are materialized.This becomes infeasible early for feature-based item towers.

4 Experiments

Under corrected, retrained evaluation, RetrievalFormer trades some warm recommendation accuracy for feature-based cold-item coverage and a shared open index. Exact full-softmax training is the largest measured accuracy lever but reaches a catalog-scale memory wall, while ANN search does not explain the remaining gap.

  • Warm accuracy: 0.3739 ± 0.0013 Recall@20 reaches 94.8% of the strongest MovieLens-1M baseline under corrected targets and five tuning rounds.The strongest baseline is DIF-SR at 0.3944 ± 0.0019; RetrievalFormer’s NDCG@20 is 0.1693 versus SASRecF’s 0.1911.
  • Warm accuracy: 0.3368 ± 0.0017 Recall@20 on MIND-large lies within 0.8–3.6% of the five strongest baselines.Short histories remove the history-context axis, and all models use exact full-catalog softmax ranking.
  • Gap analysis: 54% and 6.9% relative Recall@20 gains result from replacing sampled InfoNCE with exact full-softmax cross-entropy on MIND-small and MovieLens-1M, respectively.The matched MovieLens-1M configuration rises from 0.3132 to 0.3349; MIND-small rises from 0.1830 to 0.2822.
  • Gap analysis: A history-window sweep explains half of the remaining MovieLens-1M gap, while further item-side factorization and teacher distillation remain future work.The residual is not explained by approximate search, embedding sharing, or the tested item-ID pathway.
  • Scale and diagnostics: ANN approximation contributes nothing to the measured accuracy gap, whereas exact full-softmax training runs out of memory at a 240K-item catalog.Full-softmax materializes the catalog-wide item-tower forward with gradients at every training step, creating memory linear in catalog size.
  • Strict item cold-start: 0.172 ± 0.006 cold Recall@20 exceeds ALDI by 1.4× and the training-free floor by 3× without dedicated cold-start training.A frozen content-plus-MF score fusion reaches 0.172 ± 0.006 cold and 0.276 ± 0.001 warm, above every dedicated baseline on both axes.

5 Limitations

The evaluation has important scope and protocol boundaries. Benchmark scale, tuning asymmetries, and limited search-side evidence constrain how broadly the results should be interpreted.

  • Evaluation scope: The model received broader configuration search than each baseline, while baselines retained a default 50-item history window.The authors did not enforce parameter parity, and did not sweep longer history windows for MovieLens baselines.
  • Evaluation scope: No head-to-head accuracy comparison uses the 105-plus-item catalog scale motivating the shared-index architecture.Accuracy grids cover 3.4K and 9.4K items, while the 240K memory-wall measurement has no accompanying accuracy result.
  • Representation scope: Feature-based item representations inherit metadata-quality bias for sparse or malformed descriptions, especially for long-tail items.The method does not correct this coverage concern.
  • Search-side evidence: The search-side cold-coverage result uses one external log, an untuned BM25 comparator, a single full-catalog seed, and limited statistical power.The evidence establishes no measurable cold penalty at that power, not that none exists.

6 Conclusion

RetrievalFormer trades some recommendation quality for an index that remains open to new items. Exact training improves quality but becomes infeasible at the catalog scales a shared index demands.

  • Conclusion: 94.8% of the strongest retrained baseline is achieved on corrected targets, while the index remains open to unseen items.The strongest-baseline comparison is reported for warm recommendation accuracy.
  • Conclusion: 1.4× the strongest retrained dedicated cold-start method is achieved, with no retrieval-stage cost regression.The openness comes with measured cold-start gains and symmetric retrieval-stage cost relative to the ID-softmax baseline.
  • Conclusion: Exact-quality training at catalog scale remains the open problem because the full objective becomes infeasible in the implementation.The paper identifies the memory wall as the unresolved trade-off.

Reproducibility Statement

The reproducibility setup uses public datasets, frozen item-text embeddings, retrained and tuned baselines, and full-catalog evaluation with verified cross-harness parity.

  • Data and models: All datasets are public: MovieLens-1M, MIND, and AliEC.The statement identifies the datasets used for evaluation.
  • Data and models: Item text is encoded by a frozen gte-Qwen2-1.5B sentence encoder with 1,536 dimensions and L2 normalization.A newer 0.6B model is used only for the encoder-substitution control.
  • Evaluation: Baselines are retrained with full-softmax cross-entropy, tuned through five documented one-lever rounds, and evaluated full-catalog unmasked.The setup also verifies cross-harness data and candidate parity directly.

A Cold-Start Protocol Details

The cold-start protocol partitions items before interaction splits, trains baselines on warm-item data with content, and evaluates all methods identically.

  • Data partitioning: 80% of items form the warm pool and 20% form the cold pool, with cold items split disjointly by item.Warm interactions are split 8:1:1 by interaction; cold interactions are split 1:1 by item.
  • Baseline training: Cold-start baselines train on the warm pool using item content, including genre and year metadata or frozen title embeddings.The protocol includes DropoutNet, Heater, and ALDI variants.
  • Baseline training: All three cold-start baselines consume the same frozen matrix-factorization backbone and undergo five documented tuning rounds.Champions are confirmed at three seeds under a 500-epoch cap with early stopping.
  • Reference floor: The last-item floor ranks candidates by cosine similarity to the frozen text embedding of the user's history.This provides the training-free reference used in the cold-start evaluation.

B Evaluation-Protocol Sensitivity

Seen-item echo can materially alter reported sequential-recommendation scores, so masking choices are central to interpreting model comparisons. The effect extends beyond MovieLens and does not depend on the target-definition artifact.

  • 15–150× random-ranking expectation is the observed echo rate on sparser MIND and AliEC benchmarks.The expectation is defined as 20× mean history length divided by catalog size.
  • Masking flips no model comparison on MIND and AliEC despite their elevated echo rates.
  • Echo is computed from input sequences alone and is independent of the target-definition artifact.
  • Both-sides seen-item exclusion lifts the NDCG@5 ratio against DIF-SR from 85% to 90%.

C Extended Analyses

Extended analyses attribute performance differences to sampled-training signal limitations, encoder capacity and protocol dependence, while identifying shared embeddings as a robustness aid rather than a guaranteed optimum driver.

  • With 128 negatives against 3,416 items, each competitor receives gradient with probability approximately 0.04 per query, versus all 3,415 under full-softmax CE.
  • 0.054 cold recall for Heater with shared frozen text embeddings is below its 0.087 baseline, while warm recall rises to 0.251 ± 0.003.DropoutNet cold recall likewise falls from 0.077 to 0.059; the control bounds reference recipes rather than architectural ceilings.
  • 0.267 MF warm-slice score versus the content tower's protocol-relative result illustrates task dependence rather than a controlled decomposition.The protocols differ in split, candidate set and checkpoints.
  • 3.4% Recall@20 is lost when dual-encoder towers use separate embedding tables, while removing shared features costs 6.9% over the tuning grid.At the single best configuration, the gain is within seed noise; shared embeddings primarily buy robustness.
Loading 2608.24079v1…