PaperlayerView exact revision on arXiv

Source-linked AI summary

INSPIRED: Toward Sociable Recommendation Dialog Systems

Shirley Anugrah Hayati, Dongyeop Kang, Qingxiaoyang Zhu, Weiyan Shi, Zhou Yu

arXiv:2009.14306v2cs.CL

TL;DR

Recommendation dialog research lacked naturally occurring human data annotated for sociable strategies and linked to recommendation success. INSPIRED addresses this gap with a 1,001-dialog movie recommendation dataset, strategy annotations, outcome measures, and strategy-aware end-to-end systems. Sociable strategies correlate with successful recommendations, while incorporating strategy annotations improves automatic and human evaluation.

  • Problem

    Recommendation dialog research lacked datasets and studies focused on human sociable strategies for persuasive, successful recommendations.

  • Method

    INSPIRED provides 1,001 natural human-human movie recommendation dialogs with social-science-based strategy annotations and recommendation outcome measures.

  • Results

    Strategy-aware end-to-end dialog models perform better than models without strategy annotations in both automatic and human evaluation.

  • Takeaways & Limitations

    Sociable strategies have a positive impact on recommendation acceptance and dialog quality, supporting sociable recommendation dialog research.

  • Takeaways & Limitations

    The authors identify dynamic strategy selection, personality-based personalization, and jointly trainable recommendation and movie-information modules as future work.

Abstract

from arXiv · show

In recommendation dialogs, humans commonly disclose their preference and make recommendations in a friendly manner. However, this is a challenge when developing a sociable recommendation dialog system, due to the lack of dialog dataset annotated with such sociable strategies. Therefore, we present INSPIRED, a new dataset of 1,001 human-human dialogs for movie recommendation with measures for successful recommendations. To better understand how humans make recommendations in communication, we design an annotation scheme related to recommendation strategies based on social science theories and annotate these dialogs. Our analysis shows that sociable recommendation strategies, such as sharing personal opinions or communicating with encouragement, more frequently lead to successful recommendations. Based on our dataset, we train end-to-end recommendation dialog systems with and without our strategy labels. In both automatic and human evaluation, our model with strategy incorporation outperforms the baseline model. This work is a first step for building sociable recommendation dialog systems with a basis of social science theories.

1 Introduction

INSPIRED addresses the lack of human-centered evidence about sociable recommendation strategies by introducing a naturally collected, strategy-annotated movie recommendation dataset. Analyses link these strategies to recommendation success, and strategy-aware dialog systems outperform a baseline in automatic and human evaluation.

  • Motivation: Existing recommendation dialog datasets lack studies of human communication strategies for persuasive, successful recommendations.Prior datasets commonly use homogeneous question-answer exchanges and show limited user engagement.
  • Dataset: INSPIRED contains 1,001 natural human-human recommendation dialogs with manual annotations of sociable strategies grounded in social science theory.The collection imposed no restrictions on the number or type of movies recommended.
  • Findings: Sociable recommendation strategies are correlated with successful recommendations in dialogs.The analysis motivates building recommendation systems that incorporate sociable behavior.
  • System evaluation: Strategy-aware end-to-end dialog systems outperform systems without strategy annotations in both automatic and human evaluation.Both systems were trained on INSPIRED, differing in whether recommendation strategies were encoded.
  • Implications: INSPIRED connects social science and computational linguistics as a basis for future sociable recommendation dialog research.The authors describe this connection as opening opportunities for future studies.

2 Related Work

Prior recommendation dialog resources largely emphasize structured or simulated exchanges rather than sociable, language-centered recommendation conversations. INSPIRED combines preference elicitation, recommendation, sociable strategy annotations, and outcome ratings to study recommendation success.

  • Social science foundations: Social science research associates credibility, likeability, friendliness, humor, and personal disclosures with persuasive recommendations.These theories motivate the sociable strategy design in INSPIRED.
  • Research gap: Existing conversational recommendation systems do not focus on sociable strategies for persuasive outcomes because suitable datasets are lacking.Table 1 compares recommendation dialog datasets across relevant factors.
  • Prior datasets: Earlier datasets commonly use template-based question-answer pairs from reviews, with recommenders repeatedly eliciting product preferences.Such collections provide structured utterances rather than broad recommendation conversations.
  • Prior datasets: Gamified recommendation dialogs may poorly reflect real-world interaction because seekers pretend to like the provided movies.This limitation is identified for GORECDIAL.
  • INSPIRED’s distinction: Unlike preference-elicitation-only datasets, INSPIRED includes both preference elicitation and recommendation, plus sociable strategy annotations and recommendation ratings.This combination supports studying how conversational strategies relate to recommendation outcomes.
  • Contribution: INSPIRED connects sociable communication strategies with recommendation success and reports their positive impact in dialog settings.The work extends prior findings about strategies in other task-oriented interactions.

3 Recommendation Dialog Collection

The authors collected INSPIRED through natural two-person movie recommendation tasks with role-specific interaction, preference elicitation, recommendation, and post-task outcome measures. The dataset includes 1,001 dialogs and records acceptance, viewing behavior, and ratings, with 79.7% successful cases.

  • Collection resources: The movie database contains 17,869 movies with trailers and metadata, supported by a search interface for recommenders.Movies were curated from Movieclips Trailer2 and MovieLens, covering releases from 2008 to 2020.
  • Participants and setting: Two randomly paired Mechanical Turk workers receive recommender and seeker roles in a more realistic collection setup.Recommenders face no movie-count limit, seekers respond according to true preferences, and trailer watching is recorded.
  • Recommendation task: Recommenders first gather movie preferences, then recommend movies while continuing to interact with seekers.Before chatting, recommenders receive tips and examples for sociable recommendation strategies.
  • Outcome measures: Seekers may accept or reject recommendations, watch all or part of an accepted trailer, skip it, and rate it on a five-point Likert scale.The collection records trailer viewing duration and post-task ratings.
  • Dataset characteristics: INSPIRED contains human annotations for sociable strategies despite having fewer samples than REDIAL or GORECDIAL.The dataset can also be combined with other datasets in a semisupervised setting.
  • Collection outcomes: 79.7% of dialogs are successful cases in which seekers accept the final recommendation.All accept and reject dialogs are used for training so systems encounter diverse responses.

4 Recommendation Strategy Annotation

INSPIRED defines sociable and preference-elicitation strategies for movie-recommendation dialogs, grounding the scheme in social-science theories. The 1,001-dialog dataset uses expert- and crowd-worker annotation with substantial agreement.

  • Strategy scheme: INSPIRED divides recommendation-related utterances into sociable strategies and preference-elicitation inquiries.The scheme was developed through content analysis and prior research on human recommendation behavior.
  • Sociable strategies: Personal opinion expresses a recommender’s subjective view of a movie’s plot, actors, or other attributes.
  • Sociable strategies: Personal experience shares the recommender’s own movie-related experience and, with personal opinion, constitutes self-disclosure linked to rapport.
  • Sociable strategies: Similarity signals empathy or like-mindedness about movie preferences, while encouragement praises the seeker’s taste or promotes the recommended movie.
  • Additional strategies: Offering help communicates an explicit intention to help or transparency, whereas preference confirmation asks or rephrases the seeker’s preferences.
  • Annotation statistics: 30% of recommendation sentences were experience inquiries, 27% encouragement, and 14% personal opinion.Recommendation is defined as suggesting a new movie title for the first time.
  • Annotation quality: Expert annotators reached Kappa 0.77, and crowd-worker majority agreement was 0.78 across dialogs.Five workers annotated each dialog, with experts resolving disagreements for quality control.

5 Strategy Analyses

The analysis examines when sociable strategies occur and whether they predict successful recommendations. Sociable strategies remain associated with success even after accounting for movie attributes.

  • Strategy distribution: Offering help and similarity occur early, credibility and personal opinion appear during persuasion, and encouragement occurs mainly in the middle and at the end.
  • Success definition: Success means watching more than 50% of a recommended trailer and rating it 4 or 5 stars; rejection or skipping indicates failure.
  • Analysis method: A logistic regression predicts recommendation success from the frequency of each strategy in a dialog.The outcome is coded 1 for successful and 0 for unsuccessful recommendations.
  • Strategy associations: Personal opinion, similarity, encouragement, and credibility have significant positive effects on successful recommendations.
  • Strategy associations: Similarity has the highest coefficient, while preference-elicitation inquiries do not significantly contribute to success.The authors still describe preference queries as crucial for understanding seekers’ tastes.
  • Movie attributes: With movie attributes included, similarity, personal opinion, and encouragement remain significant, whereas no movie attribute significantly affects success.The reported strategy associations have p < 0.05.

6 Recommendation Dialog Systems

The paper compares a baseline end-to-end dialog model with a strategy-supervised model built on INSPIRED. Prepending strategy tokens improves automatic performance, and human users prefer the strategy model across evaluation aspects.

  • Experimental setup: The dataset is split into 801/100/100 dialogs for training, validation, and testing, with TMDB5 heuristics selecting movies.
  • Model design: The models use separate Transformer-based language models for seeker and recommender utterances, trained in alternating order.The strategy-incorporated model prepends a strategy as a special input token; the baseline omits it.
  • Strategy control: The system generates five candidate sentences and selects one containing encouragement or, otherwise, the longest candidate.Among tested combinations, the encouragement-only model performed best.
  • Supporting classifier: The recommendation classifier achieves 95.4% accuracy and 91.2% F1-score for identifying recommendation utterances.
  • Experimental comparison: The experiment compares a baseline dialog model without strategy supervision against a strategy-supervised model using automatic metrics and human evaluation.
  • Results: Prepending strategies improves perplexity and BLEU, while human users prefer the strategy model over the baseline on all five evaluated aspects.The aspects are fluency, consistency, naturalness, persuasiveness, and engagingness.

7 Conclusion and Future Work

INSPIRED introduces a natural-setting recommendation-dialog dataset annotated with sociable strategies and links those strategies to recommendation outcomes and dialog quality. The authors identify future directions for more dynamic, personalized, and jointly trainable systems.

  • INSPIRED is a recommendation-dialog dataset collected in a natural setting and annotated with sociable recommendation strategies.
  • The authors analyze connections between sociable strategies and recommendation results, finding positive effects on recommendation acceptance and dialog quality.
  • Future work will explore dynamic strategy selection after decoding and strategy patterns for users with different personalities and movie preferences.
  • The authors also plan to extend the model with jointly trainable movie-recommendation and movie-information modules.

A Movie Trailer Database Creation

The system constructs a trailer-enriched movie database and uses heuristics to select recommendations from dialog context. Its dialog model represents each turn as seeker and recommender utterance probabilities conditioned on conversation history.

  • Movie database: Movie records are enriched with YouTube trailers and metadata such as plots, actors, and genres using the OMDB API.
  • Movie database: Trailers are restricted to under five minutes, and movies without retrieved trailers are removed from the database.
  • Movie database: The database links MovieLens movies and user reviews for future recommendation-system research.
  • Recommendation heuristics: For cold-start cases, the system recommends the most recent movie matching the seeker’s last mentioned genre when a recommendation is generated.
  • Recommendation heuristics: When a seeker has mentioned a movie, the system queries the last positively or neutrally viewed movie and selects the first unserved recommendation.
  • Dialog model: The dialog model factorizes each turn into probabilities for seeker and recommender utterances conditioned on prior dialog history.

D Training Set-up

The training setup uses GPT-2 with delexicalized movie information and strategy labels, compares baseline and strategy-incorporated models, and applies sampling during inference. The strategy model generates a strategy before conditioning utterance generation on it.

  • Model and optimization: The models use GPT-2 small with 117M parameters, a 12-layer, 12-head Transformer, and a 768-dimensional hidden size.
  • Input representation: Movie attributes and strategy labels are represented as special tokens, with real movie information appended to the utterance.
  • Training: The models are pretrained on ReDial and movie-preference-elicitation datasets before training on INSPIRED.
  • Training: The baseline usually converges after two epochs, whereas the strategy-incorporated model usually converges after three.
  • Inference: Inference combines top-k and top-p sampling, with manually tuned temperature, p, and k parameters.
  • Strategy incorporation: The strategy-incorporated model generates a strategy first and then generates an utterance conditioned on that strategy.

E Additional User Study

An additional user study asked participants to rate each model independently on five conversational qualities. The study complements the comparative evaluation with interactive single-model ratings.

  • Participants rated fluency, consistency, naturalness, persuasiveness, and engagingness on a five-point scale.
  • For each model, 25 participants interacted with it, yielding 50 users in total.
  • Unlike the comparative study, each participant interacted with only one model because comparison was not required.

H Dialog Collection and Annotation Interfaces

The paper documents interfaces for collecting human-human recommendation dialogs, annotating recommendation strategies, and evaluating human-system interactions. Figures and tables illustrate the participant workflow, annotation process, rating criteria, and example dialogs.

  • Dialog annotation: Figures 12 and 13 show separate interfaces for defining annotation strategies and annotating dialogs by crowd-workers.Figure 6, 7, 9, and 11 are identified as dialog-collection interfaces, whereas Figures 12 and 13 support annotation.
  • Evaluation: Human ratings use five-point Likert scores for fluency, consistency, naturalness, persuasiveness, and engagingness.Fluency and consistency were not measured for the human recommender because the human-human dialogs preceded the user study.
  • Dialog examples: Tables 11 and 12 provide labeled human-human dialog examples, while Tables 13 and 14 compare example interactions involving baseline and strategy-incorporated systems.REC denotes Recommender or recommendation system, REC SYS identifies the system speaker, and SEEK denotes Seeker.
  • Dialog collection: The dialog-collection workflow includes onboarding, recommender strategy tips, preference elicitation, recommendation with search, and seeker acceptance or rejection.The interfaces cover both participant roles and the final transfer decision.
  • Recommendation outcome: Figure 11 presents a page where the seeker can watch the recommended video after the recommendation is sent.This page follows the interface in which the recommender sends the movie and the seeker chooses whether to accept or reject it.
Loading 2009.14306v2…