Source-linked AI summary
You Get What You Chat: Using Conversations to Personalize Search-based Recommendations
Ghazaleh Haratinezhad Torbati, Andrew Yates, Gerhard Weikum
TL;DR
Personalized search-based recommendation has often used explicit signals, while chats offer another source of information about users’ interests and tastes. The paper uses chat-derived models to re-rank entity results and compares them with questionnaire profiles, finding no clear overall winner.
Problem
The paper investigates whether online chats can provide implicit interest signals for personalizing search-based entity recommendations, a setting where chats had not previously been explored.
Source: §1 · p. 2Method
The authors use chat-derived user models to personalize re-ranking of entity results, adding domain-specific vocabulary and entity-based expansion, and compare them with concise questionnaire profiles.
Source: §1 · pp. 2–3, §2.1 · p. 5Results
Chat- and questionnaire-based re-rankings both substantially improved over original search-engine rankings and query-only re-rankings, with no clear winner between them overall.
Source: §8 · p. 12Takeaways & Limitations
Questionnaires are transparent and adjustable but require user effort, whereas chats require no effort but yield less transparent models and carry higher privacy risks.
Source: §8 · pp. 12–13Chat-derived models are less transparent and not easily adjustable by users, and chat data carries higher privacy risks.
Source: §8 · p. 13
Abstract
from arXiv · showhide
Prior work on personalized recommendations has focused on exploiting explicit signals from user-specific queries, clicks, likes, and ratings. This paper investigates tapping into a different source of implicit signals of interests and tastes: online chats between users. The paper develops an expressive model and effective methods for personalizing search-based entity recommendations. User models derived from chats augment different methods for re-ranking entity answers for medium-grained queries. The paper presents specific techniques to enhance the user models by capturing domain-specific vocabularies and by entity-based expansion. Experiments are based on a collection of online chats from a controlled user study covering three domains: books, travel, food. We evaluate different configurations and compare chat-based user models against concise user profiles from questionnaires. Overall, these two variants perform on par in terms of NCDG@20, but each has advantages in certain domains.
1 Introduction
The paper treats user-to-user chats as an implicit source of taste signals for re-ranking medium-grained entity searches. It investigates chat-based personalization, its comparison with concise profiles, and domain- and entity-aware model enhancements.
1 Introduction
- Search-based recommendation re-ranks entity results for medium-grained queries according to a model of the user's interests and preferences.The setting lies between coarse queries such as “love novels” and fine-grained queries such as “similar to Stephenie Meyer’s Twilight.”Source: §1 · p. 1
- Chats had not been explored for search-based product-entity recommendation beyond work leveraging product reviews, motivating the paper's focus on this implicit signal.The paper identifies this as the first study of chats for search-based recommendation.Source: §1 · p. 2
- The study asks how chat-based personalization compares with concise profiles and how domain customization and entity awareness affect user models.Its questions cover books, food recipes, and travel destinations.Source: §1 · p. 2
- The paper compares chat-derived models with concise questionnaire profiles, which offer users interpretable and adjustable representations but provide less information than behavior signals.The paper frames chats as rich signals of individual interests and tastes that users can selectively restrict for privacy reasons.Source: §1 · p. 2
- The approach adds per-domain vocabulary weighting and entity-based expansion, and releases questionnaires, pairwise chats, document URLs, and user search-result assessments for three domains.The dataset covers books, travel, and food.Source: §1 · pp. 2–3
2 Computational Model and Re-ranking
The paper re-ranks an initial pool of non-personalized entity results using query and user models. It explores language-model, BM25, and KNRM ranking, with user-model inputs and domain-specific term weights.
2 Computational Model and Re-ranking
- The computational setting re-ranks initial results using three ranking families and a user model derived from either chats or a questionnaire.Queries are medium-grained, and documents are entity-level answers from sites covering books, travel, and food; the user models are bags of words or n-grams.Source: §2 · p. 3
2.1 Re-Ranking Methods
- The language-model method combines query-document and user-document Kullback–Leibler divergences in a linear mixture, with Dirichlet smoothing and domain-specific term weights.Unigrams outperformed bigrams and trigrams in pilot experiments.Source: §2.1 · pp. 3–4
- BM25 incorporates the user model by expanding the query with user-model terms, while domain-specific weights can focus or filter those contributions.The paper discusses reducing noise and focusing the expanded query in later sections.Source: §2.1 · pp. 4–5
- KNRM scores query-document relevance by comparing embedding similarities between query and document terms, and incorporates the user model through query expansion.During training, KNRM learns how to weight different embedding-similarity levels.Source: §2.1 · p. 5
3 Domain-specific Vocabulary Weighting
Domain-specific vocabulary weighting adjusts term contributions to reflect their informativeness for books, travel, or food. The paper estimates term weights from retrieved documents and finds no significant empirical change from user-specific vocabulary weighting.
3 Domain-specific Vocabulary Weighting
- Domain-specific weighting distinguishes terms that signal interests in a particular domain from terms that are less informative there, even when their idf values are comparable.For travel, “history” and “museum” are presented as useful cues, unlike “price” and “bargain.”Source: §3 · p. 5
- Term weights estimate the probability that a term occurs in a domain-specific document or chat given that it occurs in the general corpus.The weights can adjust or eliminate term contributions in the ranking models.Source: §3 · p. 5
- Using chats by the same user rather than a universal corpus for user-specific vocabulary weighting did not significantly change empirical results, so the paper disregards that variant.The domain estimator uses each domain’s retrieved documents against the documents for all three domains together.Source: §3 · p. 5
4 Entity Expansion
The paper identifies entities and concepts in chats and questionnaires, then expands user models with descriptive text while filtering entities by domain relevance. Crowdsourced markup addresses recognition errors caused by colloquial chat language.
4 Entity Expansion
- Standard named-entity recognition and disambiguation linked chat and questionnaire spans to YAGO, but colloquial language caused many recognition errors.Crowdsourced marking of entities and general concepts eliminated nearly all NER errors; sampled NED precision was approximately 0.83.Source: §4 · pp. 5–6
- The paper expands detected entities with terms from their Wikipedia descriptions rather than adding entity names directly or using noisy entity embeddings.The descriptions provide content sketches for books and highlights for travel destinations, and the resulting terms are added to general and domain-specific user models.Source: §4 · p. 6
- Selective expansion filters candidate entities by their similarity to a domain model to avoid poor or misleading cues for the target domain.The domain models are weighted averages of the 50 most related words and entities for the relevant Wikipedia articles.Source: §4 · p. 6
- The authors tune the domain-model size and similarity thresholds by grid search to maximize area under entity-detection and disambiguation precision-recall curves.Manually annotated entities in domain-specific questionnaires serve as ground truth for domain relatedness.Source: §4 · pp. 6–7
5 Data Collection
A four-week study with 14 students collected chats and questionnaires about books, travel, and food to build the paper’s user data.
5 Data Collection
- Fourteen students participated in a four-week study, chatting in randomly assigned pairs three times per week.The first week’s chats were general; later chats focused on books, travel, and food.Source: §5 · p. 7
- In a four-week study of 14 students, each user averaged 2.8 sessions per domain and about 11 sessions overall, with averages of 653 utterances and 3,934 tokens per user.Source: §5 · p. 7
- Users completed an 18-question general questionnaire and separate questionnaires with 2 book, 5 travel, and 10 food questions.The general questions covered demographics, interests, and personality.Source: §5 · p. 7
6 Experimental Studies
The experiments find that personalization improves NDCG@20 over query-only ranking, while domain weighting has marginal effects and entity expansion yields small gains for chat-derived models.
6.1 Setup
- The study assessed 75 queries (25 per domain), 90 or more entity answers per query, and 2,673 graded judgments across 113 user-query pairs; its primary metric was NDCG@20.Compared methods included LM, LM-embed, BM25, KNRM variants, and a commercial search-engine ranking.Source: §6.1 · pp. 7–8
6.2 User Models: None vs. Chats vs. Questionnaires (RQ0 and RQ1)
- LM-embed did not improve over LM, and KNRM variants underperformed Query-Only, likely because training data and input size were limited.The authors attribute the embedding result to crude term relatedness diluting query focus.Source: §6.2 · p. 9
- Questionnaire models performed slightly better than chat models overall, but differences were minor; LM chats-Dom reached 0.822 versus 0.811 for chats-All (p=0.0018).The strongest variants generally used domain-specific inputs.Source: §6.2 · pp. 9–10
- Personalized user models improved NDCG@20 over query-only rankers by 2 to 4 percentage points, with most gains statistically significant.Results averaged 113 user-query pairs; Table 2 marks significant improvements over Query-Only with asterisks.Source: §6.2 · p. 8, Table 2 · p. 9
- For books, chat models significantly improved over questionnaire models; travel favored Dom questionnaires, while food had lower NDCG@20 than the other domains despite personalization gains.For food, Dom+Gen questionnaire profiles improved up to 2% over the search-engine results.Source: §6.2 · p. 10
- Precision@1 gains approached 10%: LM rose from 70% with query-only to 81% with questionnaires, and BM25 rose from 66% to 83%.Personalized reranking also outperformed the search-engine baseline on NDCG@top10, reaching around 83% overall and up to 87% for travel.Source: §6.2 · p. 10
6.3 Domain Vocabularies (RQ2)
- Domain-specific vocabulary weighting produced small, marginal gains that were not statistically significant (p-value > 0.1).The authors suggest chats’ thematic diversity may leave them insufficiently focused on domain-specific topics.Source: §6.3 · p. 10
6.4 Entity Expansion (RQ3)
- Entity expansion improved chat-based user models by about 1% with statistical significance (p-values < 0.05), while questionnaire models rarely improved significantly.The authors attribute the questionnaire result to those profiles already being concise and based on high-quality inputs.Source: §6.4 · p. 11
7 Related Work
Prior work personalizes search rankings from behavior histories and uses text-based models in other recommendation settings. Entity search and user-to-user chats remain less explored in the cited work.
7 Related Work
- Text-based recommenders have been studied in e-learning, literature exploration, and tourism, but the cited systems are not query-based.They use rich user-history and interest models while disregarding search results returned on the user’s behalf.Source: §7 · p. 11
- Personalized ranking has used queries and browsing histories, while language models and session signals support ranking; none of these cited approaches targets chat-derived entity-search models.Interactive recommenders instead use dialogs between users and systems, not conversations among users.Source: §7 · pp. 11–12
- Personalized entity search has been explored only in limited settings, including book recommendation and product search based on user-review embeddings.The product-search approach uses reviews on product pages, unlike this paper’s user-user chats.Source: §7 · p. 12
- Query expansion has drawn on user-provided tags, email histories, and query-history-based personalized embeddings.These cited approaches use sources other than user-user chats.Source: §7 · p. 12
8 Conclusion
Chat-based and questionnaire-based user models both improve on original search-engine rankings and non-personalized query-only rerankings, with no clear winner between them. They offer different trade-offs in effort, transparency, adjustability, and privacy.
8 Conclusion
- Both chat-based and questionnaire-based models substantially improve over original search-engine rankings and non-personalized query-only rerankings.The paper presents this as the first exploration of user-to-user conversations for personalizing search-based recommendations.Source: §8 · p. 12
- There is no clear winner between chat-based and questionnaire-based rerankings; each user-model paradigm has specific benefits.The paper compares the two approaches rather than selecting one overall winner.Source: §8 · p. 12
- Questionnaires are transparent and scrutable but require user effort, and few users seem ready to update them periodically as interests evolve.Most users seem fine with a one-time questionnaire.Source: §8 · p. 12
- Chats require no user effort and can update without intervention, but their models are less transparent, harder to adjust, and pose higher privacy risks.Domain awareness and entity expansion improve NDCG only slightly; representing detected entities explicitly is a step toward more scrutable chat profiles.Source: §8 · p. 13