PaperlayerView exact revision on arXiv

Source-linked AI summary

Extracting Interaction-Aware Monosemantic Concepts in Recommender Systems

Dor Arviv, Yehonatan Elisha, Oren Barkan, Noam Koenigstein

arXiv:2511.18024v1cs.IRcs.AIcs.LG

TL;DR

Recommender embeddings are difficult to interpret, and existing language-model SAE methods do not account for the interactions between separate user and item embeddings. The paper trains a prediction-aware Sparse Autoencoder through a frozen recommender, preserving affinity predictions while extracting concepts. Across two models and two datasets, the resulting neurons capture properties including genres, popularity, and temporal trends.

  • Problem

    Recommender embeddings typically lack clear semantic meaning, and language-model SAE methods do not sufficiently preserve the user–item interactions that drive recommendation quality.

    Source: Introduction · p. 1, Related Work · p. 2

  • Method

    A prediction-aware Sparse Autoencoder reconstructs embeddings while backpropagating through a frozen recommender to preserve predicted user–item affinities.

    Source: Introduction · p. 1, Loss Functions · p. 2

  • Results

    Across two common recommender models and two datasets, unsupervised neurons consistently capture high-level properties such as genres, popularity, and temporal trends.

    Source: Conclusion · p. 7

  • Takeaways & Limitations

    Selective latent-dimension changes support post hoc content exposure, genre suppression, and targeted item promotion without retraining the base model.

    Source: Introduction · p. 1

    Neuron purity can decline at higher K when labeling noise, genre ambiguity, or missing annotations blur concept boundaries.

    Source: Quantitative Analysis · p. 6

Abstract

from arXiv · show

We present a method for extracting \emph{monosemantic} neurons, defined as latent dimensions that align with coherent and interpretable concepts, from user and item embeddings in recommender systems. Our approach employs a Sparse Autoencoder (SAE) to reveal semantic structure within pretrained representations. In contrast to work on language models, monosemanticity in recommendation must preserve the interactions between separate user and item embeddings. To achieve this, we introduce a \emph{prediction aware} training objective that backpropagates through a frozen recommender and aligns the learned latent structure with the model's user-item affinity predictions. The resulting neurons capture properties such as genre, popularity, and temporal trends, and support post hoc control operations including targeted filtering and content promotion without modifying the base model. Our method generalizes across different recommendation models and datasets, providing a practical tool for interpretable and controllable personalization. Code and evaluation resources are available at https://github.com/DeltaLabTLV/Monosemanticity4Rec.

Introduction

Recommender embeddings can be accurate yet opaque, and methods developed for language models do not preserve the explicit user–item interactions central to recommendation. The paper introduces a prediction-aware Sparse Autoencoder framework for extracting interpretable concepts while preserving those interactions and enabling post hoc interventions.

Introduction

  • Recommender embeddings often lack clear semantic meaning, limiting interpretability despite their role in scalable, personalized recommendations.The paper also links this opacity to reduced user trust and challenges for debugging, fairness, and accountability.Source: Introduction · p. 1
  • Unlike language models, recommenders derive relevance from interactions between separate user and item embeddings, which existing language-model SAE methods do not adequately preserve.These interaction patterns drive recommendation quality.Source: Introduction · p. 1
  • The proposed SAE uses a prediction-aware reconstruction loss that backpropagates through a frozen recommender to align reconstructed embeddings with predicted user–item affinities.It replaces Top-K sparsity with KL-divergence regularization, which the authors say eliminates dead neurons and improves representation stability.Source: Introduction · p. 1
  • Selective changes to latent dimensions support post hoc exposure to novel content, suppression of undesired genres, and promotion of items to targeted audiences without retraining the base model.The approach integrates into two-tower architectures without architectural changes or additional supervision.Source: Introduction · p. 1
  • The paper presents applications in content filtering, preference steering, and targeted item promotion, alongside an evaluation across multiple recommender models and diverse datasets.The introduction lists these as contributions of the work.Source: Introduction · p. 1

Related Work

Prior SAE-based monosemanticity work primarily studies latent spaces in language models, while recommender systems derive relevance from interactions between separate user and item embeddings. Existing recommender interpretability approaches either design model-specific factors or explain outputs, leaving latent embedding semantics opaque.

Monosemanticity via Sparse Autoencoders

  • SAEs have been used to uncover interpretable latent dimensions in language models, with related work addressing feature absorption, feature hedging, and neuron-level monosemanticity metrics.Matryoshka SAEs encourage coarse-to-fine specialization, while very large dictionaries can produce redundant features.Source: Related Work · p. 2
  • Prior SAE research largely targets transformer language-model representations and does not address semantics arising from explicit interactions between separate user and item embeddings.The paper identifies this interaction-based structure as an un explored challenge for recommender systems.Source: Related Work · p. 2

Interpretability in Two-Tower Recommender Architectures

  • Two-tower recommenders combine independently encoded users and items, but existing model-specific factors, attention weights, and post-hoc explanations leave latent embedding semantics opaque.Post-hoc approaches use feature attribution and perturbation analysis to explain outputs.Source: Related Work · p. 2

Method

The method applies a Sparse Autoencoder to user and item embeddings in two-tower recommenders, using an affinity-aware objective to preserve interaction semantics. It trains post hoc against a frozen recommender with reconstruction and sparsity losses.

Method

  • The framework extracts monosemantic concepts from user and item embeddings in two-tower recommender architectures.Source: Method · p. 2

General Recommender System Formulation

SAE Architecture

  • The SAE encodes embeddings into sparse latent vectors and reconstructs them; its Matryoshka structure trains nested dictionaries from general to finer-grained features.Source: SAE Architecture · p. 2

Loss Functions

  • The reconstruction objective combines embedding-level fidelity with a prediction-level loss that aligns reconstructed and original user–item affinities.The prediction-level term is the mean squared difference over sampled user–item pairs and encourages preservation of interaction semantics and ranking consistency.Source: Loss Functions · p. 2
  • Sparsity is encouraged through ℓ1 regularization and a KL penalty on bottleneck activations, with the KL term promoting low activation rates and compact representations.Source: Loss Functions · p. 3

Training Procedure

  • The SAE is trained post hoc with the base recommender frozen, while prediction-loss gradients pass through the recommender to align latent representations with its behavioral outputs.Source: Training Procedure · p. 3, Figure 1 · p. 3

Results

Across recommender models and domains, the SAE yields interpretable neurons for genres, popularity, and stylistic eras, with high semantic purity and post hoc intervention use cases. The Matryoshka experiments suggest that Last.FM neurons can organize from broad audiences to niche preferences.

Experimental Setup

  • Experiments use MF and NCF on MovieLens 1M and Last.FM, with separate SAEs and bottleneck sizes chosen for each model–dataset pair.The bottleneck has 22 neurons for ML1M and 70 for Last.FM; neuron labels are assigned by GPT-4.5 from top-activating items.Source: Experimental Setup · pp. 3, 5

Qualitative Analysis

  • Genre-aligned neurons show selective activation across movies and artists, while other neurons capture popularity and temporal or stylistic patterns.Examples include electronic-music tags in Last.FM and neurons aligned with distinct movie eras.Source: Qualitative Analysis · p. 5

Beyond Interpretability: Practical Use Cases

  • Neuron-level edits support post hoc promotion, suppression, or diversification of recommendations without retraining the base model.Examples include promoting Bob Dylan to users who prefer Metal, Contemporary Pop, or Electronic music, and steering users toward or away from selected movie genres.Source: Beyond Interpretability: Practical Use… · p. 6

Hierarchical Structure with Matryoshka

  • Matryoshka SAEs have limited impact on ML1M, while Last.FM shows broad-audience features earlier and narrower micro-genres and hybrid scenes later.The authors suggest this hierarchy may reveal layered audience structure for more controllable personalization.Source: Hierarchical Structure with Matryoshka · p. 7

Conclusion

The framework extracts unsupervised monosemantic concepts from recommender embeddings while preserving the user–item interaction patterns underlying recommendation behavior. Across two common recommender models and two datasets, the concepts capture high-level properties, while neuron-level interventions remain preliminary.

Conclusion

  • Across two common recommender models and two datasets, unsupervised neurons consistently capture high-level properties such as genres, popularity, and temporal trends.The framework uses a prediction-aware reconstruction objective to preserve user–item interaction patterns.Source: Conclusion · p. 7
  • The framework defines monosemantic neurons as latent dimensions aligned with coherent and meaningful concepts in recommender embeddings.Its prediction-aware reconstruction objective preserves the user–item interaction patterns underlying recommendation behavior.Source: Conclusion · p. 7
  • Neuron-level interventions lightly adjust recommendations, making controllability a preliminary direction for future work.The paper also suggests semantically grounded latent factors may help enable faithful explainability methods for recommender systems.Source: Conclusion · p. 7
  • The work aims to contribute to more semantically aligned and trustworthy recommender technologies as recommenders shape information access and cultural exposure.The authors identify transparency, interpretability, and accountability as increasingly important.Source: Conclusion · p. 7
Loading 2511.18024v1…