PaperlayerView exact revision on arXiv

Source-linked AI summary

What does BERT know about books, movies and music? Probing BERT for Conversational Recommendation

Gustavo Penha, Claudia Hauff

arXiv:2007.15356v2cs.IR

TL;DR

It is important to understand what pre-training teaches language models because their stored knowledge may inform conversational recommender systems. The authors probe BERT’s genre predictions and its matching of search and recommendation inputs, then study conversational recommendation with knowledge infused through multi-task fine-tuning. BERT extracts genres for 30–50% of items in its top-5 predictions, depending on domain, and has about 17% more content-based (search) than collaborative-based (recommendation) knowledge.

  • Problem

    It is important to understand what pre-training teaches language models because their stored knowledge may inform conversational recommender systems.

    Source: §1 · p. 1

  • Method

    The authors probe BERT’s genre predictions and its matching of search and recommendation inputs, then study conversational recommendation with knowledge infused through multi-task fine-tuning.

    Source: §1 · p. 2

  • Results

    BERT extracts genres for 30–50% of items in its top-5 predictions, depending on domain, and has about 17% more content-based (search) than collaborative-based (recommendation) knowledge.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The findings provide insights into what BERT can do with stored knowledge for conversational recommendation, where it fails, and how knowledge can be infused.

    Source: §6 · p. 9

    User studies are still needed to investigate how well BERT-based conversational recommendation works in practice.

    Source: §6 · p. 9

Abstract

from arXiv · show

Heavily pre-trained transformer models such as BERT have recently shown to be remarkably powerful at language modelling by achieving impressive results on numerous downstream tasks. It has also been shown that they are able to implicitly store factual knowledge in their parameters after pre-training. Understanding what the pre-training procedure of LMs actually learns is a crucial step for using and improving them for Conversational Recommender Systems (CRS). We first study how much off-the-shelf pre-trained BERT "knows" about recommendation items such as books, movies and music. In order to analyze the knowledge stored in BERT's parameters, we use different probes that require different types of knowledge to solve, namely content-based and collaborative-based. Content-based knowledge is knowledge that requires the model to match the titles of items with their content information, such as textual descriptions and genres. In contrast, collaborative-based knowledge requires the model to match items with similar ones, according to community interactions such as ratings. We resort to BERT's Masked Language Modelling head to probe its knowledge about the genre of items, with cloze style prompts. In addition, we employ BERT's Next Sentence Prediction head and representations' similarity to compare relevant and non-relevant search and recommendation query-document inputs to explore whether BERT can, without any fine-tuning, rank relevant items first. Finally, we study how BERT performs in a conversational recommendation downstream task. Overall, our analyses and experiments show that: (i) BERT has knowledge stored in its parameters about the content of books, movies and music; (ii) it has more content-based knowledge than collaborative-based knowledge; and (iii) fails on conversational recommendation when faced with adversarial data.

1 INTRODUCTION

The introduction asks what recommendation knowledge off-the-shelf BERT stores about books, movies, and music, distinguishing item-content matching from collaborative co-consumption. It probes that knowledge and then evaluates BERT for conversational recommendation, including under adversarial negatives.

1 INTRODUCTION

  • Conversational recommender systems can elicit preferences, explain suggestions, and process user feedback, while natural-language approaches have gained attention alongside voice-based agents.The introduction motivates studying what pre-training learns because BERT and other language models can store factual knowledge in their parameters.Source: §1 · pp. 1–2
  • The study asks what off-the-shelf BERT knows about recommendation items in books, movies, and music, domains chosen for their popularity.It distinguishes content-based knowledge—matching item titles to descriptions or genres—from collaborative-based knowledge—matching items consumed together.Source: §1 · p. 2
  • Genre prompts probe BERT’s masked-token predictions, while sentence similarity and next-sentence prediction compare relevant with non-relevant search and recommendation inputs without fine-tuning.Relevant recommendation pairs use frequently co-consumed items; search pairs match item titles with reviews.Source: §1 · p. 2
  • BERT stores both content-based and collaborative-based knowledge, but content-based knowledge is stronger; adversarial negatives reduce downstream nDCG@10 from 0.81 to 0.06.The probes also find genre tokens in the top five for 30%–50% of cases depending on domain, and report improved conversational recommendation from multi-task knowledge infusion.Source: §1 · p. 2
  • The second research question asks how to infuse content-based and collaborative-based knowledge into BERT for conversational recommendation.The authors hypothesize that stronger search and recommendation probe performance may correspond to better conversational recommendation.Source: §1 · p. 2

2 RELATED WORK

The related work frames conversational search and recommendation as an open challenge and reviews research on probing and adapting pre-trained language models. The paper applies those perspectives to recommendation, search, and genre probes, then tests knowledge infusion for conversational recommendation.

Conversational Search and Recommendation.

  • A full-blown conversational search and recommendation system remains a major open challenge, despite conceptual work and techniques for related tasks.Conversational systems can clarify, elicit, and refine information needs and provide recommendations, explanations, and answers [10].Source: Conversational Search and Recommendation · pp. 2–3
  • The paper examines BERT’s potential for conversational recommendation, focusing on where it excels and requires improvement.Its movie-task examples distinguish co-consumption recommendation from review-title search and genre-title matching.Source: Conversational Search and Recommendation · p. 3, Table 1 · p. 3

Pre-trained Language Models: Probing and Knowledge Infusion.

  • Prior work probes what language models encode through attention analysis, probing tasks, and diagnostic datasets, while factual-knowledge extraction has used cloze prompts [27].Some probing tasks requiring reasoning have exposed failures, including four of eight tasks for BERT and RoBERTa [42].Source: Pre-trained Language Models: Probing… · p. 3
  • Knowledge infusion approaches include intermediate tasks, domain-specific continued pre-training, multi-task learning, regularization, and adapters, with catastrophic forgetting a concern [16].The paper probes recommendation, search, and genre knowledge and studies conversational recommendation before and after multi-task knowledge infusion.Source: Pre-trained Language Models: Probing… · p. 3

3 METHOD

This section introduces BERT before describing the three probing tasks and the conversational recommendation downstream task.

3 METHOD

  • The section first briefly describes BERT, then introduces genre, search, and recommendation probes, before turning to conversational recommendation.These are the three probing tasks and the downstream task named in the section roadmap.Source: §3 · p. 3
  • Genre, search, and recommendation are presented as distinct probing tasks in the method overview.The overview places the conversational recommendation task after those probes.Source: §3 · p. 3
  • Conversational recommendation is identified as the downstream task that follows the probe descriptions.The section roadmap first covers BERT and the three probes.Source: §3 · p. 3

3.1 Background: BERT

BERT learns representations from both left and right context and was pre-trained with masked language modelling and next sentence prediction. Its pre-training data included English Wikipedia and BookCorpus.

3.1 Background: BERT

  • BERT conditions on both left and right context at every layer when learning textual representations.The model was pre-trained on English Wikipedia and BookCorpus, which contains 11,000 books.Source: §3.1 · p. 4
  • For masked language modelling, BERT replaces 15% of tokens with [MASK] and learns to predict those tokens.The task supplies the model with masked-token prediction training.Source: §3.1 · p. 4
  • For next sentence prediction, BERT classifies whether sentence B follows sentence A, with random sentences used for half the pairs.The [CLS] token is used for classification, and [SEP] separates packed sentence pairs.Source: §3.1 · p. 4

3.2 Genre Probes

Genre probes test whether BERT can recover item genres from masked-token prompts, with different templates varying how much item information is supplied.

3.2 Genre Probes

  • The MLM probe masks an item’s genre in a prompt and tests whether BERT ranks the correct genre tokens highly.The prompts pair item titles with genre information from knowledge sources.Source: §3.2 · p. 4
  • Three templates vary the input from domain only, to title plus domain, to title, domain, and an explicit genre phrase.The templates are TP-NoTitle, TP-Title, and TP-TitleGenre.Source: §3.2 · p. 4
  • The probe treats correct genre tokens ranked first as evidence of stored knowledge and reports recall at ranks 1 and 5.Because the manually selected templates are not exhaustive, their results are a lower bound on BERT’s stored knowledge.Source: §3.2 · p. 4

3.3 Recommendation and Search Probes

The paper uses two no-fine-tuning probes to test whether BERT ranks relevant items above non-relevant ones in search and recommendation. Their prompts represent user-side queries and candidate documents, with task-specific relevance construction.

3.3 Recommendation and Search Probes

  • SIM and NSP probe ranking without fine-tuning, using BERT representations for similarity or its Next Sentence Prediction head to assess sentence relationships.Queries represent user-side input, while documents represent possible system answers; non-relevant documents are randomly sampled.Source: §3.3 · p. 4

3.3.1 Probe Based on Similarity (SIM).

  • SIM ranks document sentences by dot similarity to a query, using either the final-layer CLS representation or the average of token representations.The two variants are SIMCLS and SIMMEAN.Source: §3.3.1 · p. 4

3.3.2 Probe Based on Next Sentence Prediction Head (NSP).

  • NSP ranks document sentences by their likelihood of being the next sentence after the query sentence.This probe also requires no fine-tuning of BERT.Source: §3.3.2 · p. 4

3.3.3 Templates and Prompt Generation.

  • Recommendation queries use a user-rated item and compare another item rated by that user with popularity-sampled unrated items; search pairs item reviews with their titles and random mismatched reviews.The prompts draw recommendation information from ratings and search information from reviews.Source: §3.3.3 · pp. 4–5, Table 2 · p. 5

3.4 Infusing Knowledge into LMs for Conversational Recommendation

The downstream task ranks response candidates for a conversational context, and the paper tests multi-task fine-tuning as a way to infuse probing-task knowledge into BERT.

3.4.1 Conversational Recommendation.

  • Conversational recommendation is framed as ranking available response candidates by relevance to the dialogue context.The setup assumes a historical dialogue corpus contains suitable replies, including coherent responses with item recommendations.Source: §3.4.1 · p. 5
  • BERT scores each candidate by encoding the dialogue utterances and response together, then applying a linear layer trained with binary cross-entropy.The response score is used as the predicted relevance score.Source: §3.4.1 · p. 5

3.4.2 Infusing Knowledge into LMs.

  • Multi-task fine-tuning interleaves conversational data with recommendation-NSP or search-NSP batches to infuse collaborative- or content-based knowledge.The authors use equal task weights and leave analyses of task order and weighting for future work.Source: §3.4.2 · p. 6

4 EXPERIMENTAL SETUP

The experiments draw on item-interaction and review datasets across books, movies, and music, alongside conversational datasets from online forums and ReDial. The setup also specifies probe volumes and model-training choices.

4.1 Data Sources

  • The item knowledge sources are GoodReads for books, MovieLens 25M plus crawled IMDb reviews for movies, and Amazon CDs and Vinyl for music.The datasets provide ratings, reviews, or genres as available across domains.Source: §4.1 · p. 6
  • The probes use 100k instances per task and domain, except movie genre probing, which uses about 60k movies; genre counts and diversity differ by domain.Books, movies, and music average 3.6, 1.8, and 1.4 genres, with 16, 20, and 284 distinct genres respectively.Source: §4.1 · p. 6
  • Conversational data comes from three recommendation subreddits and ReDial; the movie identifiers in ReDial enable adversarial examples, while the music forum has fewer users than the book forum.The music suggestions subreddit has 10k users compared with 292k in booksuggestions.Source: §4.1 · p. 6, Table 3 · p. 6

4.2 Implementation Details

  • Dialogue evaluation generates 50 candidate responses per context with BM25, while BERT uses large cased models and validation nDCG for early stopping.BERT is fine-tuned with Adam at a learning rate of 5e-6; comparisons include QL, QL with RM3, DAM, and MSN.Source: §4.1 · p. 6, §4.2 · p. 6

5 RESULTS

BERT retrieves genre, search, and recommendation information from its pretrained parameters, with stronger content-based than collaborative-based knowledge. Fine-tuning performs well for conversational recommendation, but adversarial candidates expose a sharp weakness; adding probe knowledge helps on ReDial.

5.1.1 Genres.

  • Without item titles, BERT still retrieves domain-related genres in the first five positions for 25%–41% of predictions.The authors interpret this as evidence that pretrained BERT contains information about genres associated with each domain.Source: §5.1.1 · p. 7
  • Title-aware genre prompts outperform domain-only prompts by 0.067–0.179 R@1, with correct genres appearing in the top five up to 50% of the time.The authors treat this as a lower bound because another prompt template might retrieve information missed by the tested prompts.Source: §5.1.1 · p. 7, Table 4 · p. 7

5.1.2 Search and Recommendation.

  • With two candidates, NSP-based BERT matches reviews to books 82%, movies 67%, and music 75% of the time, and selects recommendation items 65% of the time.NSP is the most effective tested probing technique; the probes perform better than a random baseline.Source: §5.1.2 · p. 7
  • BERT shows about 17% more content-based search knowledge than collaborative-based recommendation knowledge, and recommendation effectiveness degrades more as candidate counts rise.The authors suggest BERT may therefore be better used as a re-ranker for a few candidates.Source: §5.1.2 · p. 7, Figure 1 · p. 7
  • Books have the highest observed search-probe effectiveness, which the authors hypothesize may reflect overlap with BookCorpus, an explanation they could not verify directly.BookCorpus was no longer publicly available for checking the proposed overlap.Source: §5.1.2 · p. 7

5.2 Infusing knowledge into BERT for Conversational Recommendation (RQ2)

  • Fine-tuned BERT improves nDCG@10 over MSN by 35% for books, 15% for movies, and 16% for music in conversational recommendation.BERT reaches up to 0.98 nDCG@10.Source: §5.2 · p. 8
  • BERT’s nDCG@10 falls from 0.81 to 0.06 on adversarial ReDial candidates that replace recommended items with randomly selected items.The test examines whether BERT identifies relevant items rather than relying only on response-language cues.Source: §5.2 · pp. 8–9
  • The recommendation probe improves BERT by 9% on adversarial ReDial; conversational-recommendation nDCG@10 gains reach up to 9%, while the search probe improves ReDialBM25 effectiveness by 1%.Source: §5.2 · p. 9

6 CONCLUSIONS

The conclusion reports that BERT stores item knowledge across books, movies, and music, with more content-based than collaborative-based knowledge. It also reports conversational-recommendation results, an adversarial weakness, and gains from multi-task fine-tuning.

6 CONCLUSIONS

  • Genre probing retrieves genres for 30–50% of items in the top five, depending on domain, while BERT has about 17% more content-based than collaborative-based knowledge.Source: §6 · p. 9
  • BERT reaches 0.9 nDCG@10 versus 0.7 for the second-best baseline on conversational recommendation, but is less effective against reasonable adversarial responses with random item recommendations.Source: §6 · p. 9
  • Multi-task fine-tuning that infuses probing-task knowledge improves conversational recommendation effectiveness by up to 9%.Source: §6 · p. 9
  • User studies and additional domains remain future work to assess practical performance and broader CRS utility.Source: §6 · p. 9
Loading 2007.15356v2…