Source-linked AI summary
Ask the GRU: Multi-Task Learning for Deep Text Recommendations
Trapit Bansal, David Belanger, Andrew McCallum
TL;DR
Text recommendation must use item content while handling sparse feedback and unseen items, but prior mappings often ignore word order. The paper trains GRU text encoders end-to-end for collaborative filtering and adds metadata prediction through multi-task learning. On scientific paper recommendation, the approach improves cold-start accuracy, with up to 34% relative improvement in Recall@50, and multi-task learning further improves performance.
Problem
Existing text recommendation mappings often use topic models or averaged word embeddings, while hybrid methods may ignore word order and face sparse feedback and cold-start items.
Method
The paper uses deep recurrent neural networks, specifically GRUs, to encode item text into vectors for collaborative filtering and trains the encoder jointly with item metadata prediction.
Results
Up to 34% relative improvement in Recall@50 was achieved for cold-start scientific paper recommendation over CTR and a word-embedding model, with multi-task learning further improving performance.
Takeaways & Limitations
Recurrent text encoders and multi-task learning provide complementary performance improvements for scientific paper recommendation.
Takeaways & Limitations
Bag-of-words representations cannot distinguish texts with similar unigram statistics but different meanings, and tags may be noisy, general, or unavailable for new items.
Abstract
from arXiv · showhide
In a variety of application domains the content to be recommended to users is associated with text. This includes research papers, movies with associated plot summaries, news articles, blog posts, etc. Recommendation approaches based on latent factor models can be extended naturally to leverage text by employing an explicit mapping from text to factors. This enables recommendations for new, unseen content, and may generalize better, since the factors for all items are produced by a compactly-parametrized model. Previous work has used topic models or averages of word embeddings for this mapping. In this paper we present a method leveraging deep recurrent neural networks to encode the text sequence into a latent vector, specifically gated recurrent units (GRUs) trained end-to-end on the collaborative filtering task. For the task of scientific paper recommendation, this yields models with significantly higher accuracy. In cold-start scenarios, we beat the previous state-of-the-art, all of which ignore word order. Performance is further improved by multi-task learning, where the text encoder network is trained for a combination of content recommendation and item metadata prediction. This regularizes the collaborative filtering model, ameliorating the problem of sparsity of the observed rating matrix.
1. INTRODUCTION
Text recommendation combines user preferences with item content, but existing approaches often ignore word order or struggle with cold-start items. This paper uses recurrent text encoders and multi-task learning, improving scientific paper recommendation, especially in cold-start settings.
- Text recommendation spans blogs, social posts, news, movies, products, and research papers, with potential to increase user engagement.
- Collaborative filtering relies on rating matrices but cannot rank unseen items, while content-based methods support new items yet ignore similarities among user preferences.
- Hybrid methods combine content and ratings, but commonly ignore word order and use bag-of-words or unsupervised topic-model objectives.
- Recurrent neural networks represent item text for collaborative filtering while preserving word order, avoiding hand-engineered features, and supporting cold-start recommendation.
- Up to 34% relative improvement in Recall@50 was achieved for cold-start recommendation over CTR and a word-embedding model, while multi-task learning significantly improved all models.
2. BACKGROUND AND RELATED WORK
The paper frames text recommendation as personalized ranking from user feedback, item text, and tags, while reviewing latent-factor and hybrid methods. It highlights cold-start limitations of item embeddings and noisy or unavailable tags, motivating learned text encoders and multitask regularization.
- Problem setting: The recommendation task ranks each user’s unrated items using item text, previous likes, and item tagging information.The setting uses implicit feedback: observed likes are marked positive, while unobserved items remain ambiguous.
- Latent factor models: Latent factor models represent users and items with K-dimensional embeddings and can be trained with weighted squared loss or ranking-based objectives.For implicit feedback, observed items typically receive higher loss weights than unobserved items, reflecting uncertainty about unobserved ratings.
- Hybrid and cold-start methods: Standard factorization cannot recommend new items without learned item embeddings, motivating hybrid collaborative-filtering methods that map item content to factors.Existing hybrid approaches commonly use a differentiable or linear function of item text or manually extracted metadata.
- Related work: Collaborative topic regression combines item-word topic modeling with user-item factorization to obtain low-dimensional item representations for unseen-item recommendation.The paper uses collaborative topic regression as a state-of-the-art comparison technique.
- Metadata and multitask learning: Tags can be noisy, incomplete, user-assigned, and unavailable for new items, limiting their reliability as input features.The paper instead motivates predicting tags as an auxiliary task, using them for training-time regularization without requiring them at test time.
- Neural recommendation: The reviewed neural collaborative-filtering approaches include denoising autoencoders, but such models cannot handle cold-start items when they operate only on sparse user-item interaction vectors.The paper positions its text-to-embedding mapping within broader deep-learning approaches for recommendation.
3. DEEP TEXT REPRESENTATION FOR COLLABORATIVE FILTERING
The paper maps item text to latent representations with recurrent encoders, combines them with item-specific embeddings, and trains the resulting model for recommendation. GRU-based architectures incorporate word order, while multi-task tag prediction provides supervised regularization for sparse recommendation data.
- Text encoders: The encoder maps a token sequence to a single vector g(Xj), enabling text-based item representations for cold-start prediction.The mapping is shared across items through a parametric model, while new cold-start items omit the item-specific embedding.
- Text encoders: Averaging word embeddings is an order-insensitive baseline equivalent to a linear bag-of-words model, whereas recurrent encoders can exploit word order.Bag-of-words models cannot distinguish sentences with identical unigram statistics but different meanings.
- GRU encoders: The proposed GRU encoder reads text sequentially and produces a representation without an unsupervised text objective, training directly through supervised recommendation.GRUs are used because they are simpler than LSTMs, have fewer parameters, and offer competitive performance.
- GRU encoders: The architecture stacks two GRU-RNN layers, uses a bidirectional first layer, pools second-layer hidden states, and combines the text encoding with an item-specific embedding.Mean pooling performs best in the reported experiments, while final-state models take longer to optimize.
- Multi-task learning: Multi-task learning trains the encoder for both recommendation and tag prediction, using tags as a coarse summary or topic signal and combining the task costs with λ.The final cost is C(θ) = λCR(θ) + (1 −λ)CT (θ).
4. EXPERIMENTS
The experiments evaluate text-based recommendation models on CiteULike under warm-start and cold-start settings using Recall@50 and related Recall@M analyses. GRU encoders and multi-task learning improve recommendation performance, with especially large gains in cold-start scenarios.
- Evaluation Methodology: Two CiteULike datasets are evaluated under warm-start and cold-start recommendation settings using held-out user-article likes.Warm-start retains at least one training like for each test item, whereas cold-start removes all likes from test-fold papers.
- Compared Methods: The comparison includes topic-model CTR, order-insensitive Embed, recurrent GRU, and their multi-task variants CTR-MTL, Embed-MTL, and GRU-MTL.CTR is also modified for tag prediction, producing the CTR-MTL baseline.
- Evaluation Metrics: Recall@50 is reported for all models, while Figure 3 examines Recall@M across different recommendation-list sizes for multi-task models.Recall is used because unobserved implicit-feedback ratings may indicate either dislike or lack of exposure.
- Cold-Start Results: 18.36% and 32.74% relative improvements are achieved by GRU over the best CTR or Embed score in cold-start recommendation on CiteULike-a and CiteULike-t, respectively.In cold-start evaluation, item representations depend solely on text because item-specific embeddings are set to zero.
- Multi-Task Results: 7.8% and 7.6% recall improvements are obtained by GRU with multi-task learning on CiteULike-a and CiteULike-t, yielding 19.30% and 34.22% gains over the best baselines.Multi-task learning uniformly improves all models, while Embed benefits by up to 8.2%.
- Warm-Start Results: GRU exceeds CTR and Embed by 8.5% and 5.3% in warm-start recommendation, while GRU-MTL achieves HR@10 of 0.098 versus 0.077 for CTR-MTL.The HR@10 comparison indicates more relevant recommendations near the top of the ranked list for GRU-MTL.
5. CONCLUSION & FUTURE WORK
The paper uses deep recurrent neural networks to map item text into vectors for collaborative filtering, with RNNs and multi-task learning providing complementary performance improvements for scientific paper recommendation.
- 5. CONCLUSION & FUTURE WORK: Deep recurrent neural networks provide vector representations of item text for collaborative filtering.This text-to-vector mapping is trained directly with gradient descent and supports multi-task learning.
- 5. CONCLUSION & FUTURE WORK: RNNs and multi-task learning each contribute complementary performance improvements for scientific paper recommendation.
- 5. CONCLUSION & FUTURE WORK: Future work will apply deep architectures to users’ data and explore multi-task objectives using multiple input modalities.