Source-linked AI summary
Dialogue Response Ranking Training with Large-Scale Human Feedback Data
Xiang Gao, Yizhe Zhang, Michel Galley, Chris Brockett, Bill Dolan
TL;DR
Open-domain dialog models can produce context-relevant replies, but relevance alone does not establish engagingness. This paper trains response-ranking models from online human feedback and combines them with a human-like scoring model; human evaluation finds the ranking method correlates better with real preferences than baselines.
Problem
Context-relevant or human-sounding replies may still be bland or unengaging, while large-scale human judgments of engagingness are difficult to obtain.
Source: §1 · p. 1Method
The authors train feedback-ranking models from online replies and likes, then combine their predictions with a human-like scoring model to rank generated responses.
Source: §1 · p. 2Results
Takeaways & Limitations
The reported human evaluation supports using the combined ranking method to rank machine-generated dialog responses by human preference.
Source: §6 · p. 9Online feedback is a noisy engagingness proxy because popularity is long-tailed and subreddit traffic, timing, and social influence can distort counts.
Source: §2.2 · p. 3
Abstract
from arXiv · showhide
Existing open-domain dialog models are generally trained to minimize the perplexity of target human responses. However, some human replies are more engaging than others, spawning more followup interactions. Current conversational models are increasingly capable of producing turns that are context-relevant, but in order to produce compelling agents, these models need to be able to predict and optimize for turns that are genuinely engaging. We leverage social media feedback data (number of replies and upvotes) to build a large-scale training dataset for feedback prediction. To alleviate possible distortion between the feedback and engagingness, we convert the ranking problem to a comparison of response pairs which involve few confounding factors. We trained DialogRPT, a set of GPT-2 based models on 133M pairs of human feedback data and the resulting ranker outperformed several baselines. Particularly, our ranker outperforms the conventional dialog perplexity baseline with a large margin on predicting Reddit feedback. We finally combine the feedback prediction models and a human-like scoring model to rank the machine-generated dialog responses. Crowd-sourced human evaluation shows that our ranking method correlates better with real human preferences than baseline models.
1 Introduction
The paper targets a gap between context-relevant replies and engaging ones, using online human feedback to train response-ranking models. Its human-evaluation results favor the proposed ranking method over baselines.
1 Introduction
- Context relevance and human-likeness do not ensure a reply is engaging, and large-scale engagingness labels are difficult to obtain through annotation.The paper motivates predicting whether a response will elicit a positive reaction from an interlocutor.Source: §1 · pp. 1–2
- The authors use replies and likes from online communities as feedback data, carefully normalizing and formulating them as engagingness labels.They note that feedback can be distorted by social influence and other confounding factors.Source: §1 · p. 2
- The paper shifts response ranking to pairwise classification to reduce possible distortion from confounding factors in feedback labels.The comparison is between pairs of responses with fewer confounding factors.Source: §1 · p. 2
- Human evaluation found the ranking method correlated better with actual human preferences than baselines, while dialog perplexity showed little predictive power for Reddit feedback.The method combines feedback prediction models with a human-like scoring model to rank machine-generated responses.Source: §1 · p. 2
2 Human Feedback
The section defines three measures of online feedback for a context–reply pair and explains why raw feedback needs careful formulation before serving as an engagingness signal. It also specifies feedback-ranking and human-versus-fake prediction tasks.
2.1 Feedback metrics
- Width counts direct replies, Depth measures the maximum dialog length after a reply, and Updown is its upvotes minus downvotes.The measures capture different feedback dimensions; Width and Depth are often closely correlated, unlike Updown.Source: Table 1 · p. 2, §2.1 · p. 3
2.2 Feedback and Engagingness
- Feedback is not a direct engagingness measure: popularity is long-tailed, subreddit traffic and comment timing confound counts, and normalization is needed.Comments posted early in a thread are more likely to attract replies or likes.Source: §2.2 · p. 3
2.3 Tasks
- The tasks rank human responses by Width, Depth, or Updown feedback and classify responses as human versus fake, using random or machine-generated examples.Source: §2.3 · p. 3
3 The DIALOGRPT Method
DIALOGRPT ranks responses using human-feedback comparisons and combines feedback-based preference with human-likeness scores for machine generations.
3.1 Problem Formulation
- Same-context response pairs are ranked by feedback, restricted to replies posted within an hour and separated by a feedback threshold; the contrastive loss raises positive and lowers negative scores.Replies with more downvotes than upvotes are excluded as positives but may serve as negatives.Source: §3.1 · pp. 3–4
3.2 Model ensemble
- Machine responses are ranked by combining preference conditional on human-likeness with a human-likeness score; annotator choices on 1K machine pairs tune weights, with five judges per pair.The optimal weights may depend on the test set and annotator instructions.Source: §3.2 · p. 4
3.3 Implementation details
- A 12-layer GPT-2 model initialized from DialoGPTmedium was trained on 133M Reddit pairs; human-vs-fake negatives include retrieval and top-k DialoGPT generations.The training, validation, and test sets do not share Reddit posts.Source: §3.3 · pp. 4–5
3.4 Baselines
- Comparators include forward/reverse dialog perplexity, BM25 keyword similarity, ConveRT context-response vector similarity, BoW feedback averages, and a length-based score.Dialog perplexity is formulated to maximize mutual information between response and context.Source: §3.4 · p. 5
4 Results
DIALOGRPT led the feedback-ranking tests and improved candidate ranking over perplexity-based methods. Across evaluations, the models transferred to related tasks and datasets, while the ensemble performed reasonably across test sets and Updown aligned best with human preference.
4.1 Predicting Human Feedback
- Questions tended to attract longer conversations, broad-audience comments more direct replies, and low-feedback comments were often less contentful.Table 4 presents the bag-of-words analysis of words associated with more or less feedback.Source: §4.1 · p. 6, Table 4 · p. 7
4.2 Human-like Classification
- Feedback models performed much better than Human-vs-Rand models at distinguishing human from generated responses, while combined training on random and generated responses worked on fake-response tests but not human-feedback ranking.The authors infer that generated responses may be less proactive or engaging than human replies, and describe the models as complementary.Source: §4.2 · p. 7
4.1 Predicting Human Feedback
- DIALOGRPT achieved the highest test performance on pairwise accuracy and Spearman’s ρ, while bag-of-words outperformed dialog-perplexity models.The dialog-perplexity models were not trained with feedback labels.Source: §4.1 · p. 6, Table 5 · p. 7
4.3 Ensembling Models
- The ensemble was not the most accurate on any test set but performed reasonably across them; human preference correlated best with Updown, though Depth and Width also contributed.Source: §4.3 · p. 7, Table 8 · p. 8
4.1 Predicting Human Feedback
- Models trained on Width and Depth feedback performed reasonably on each other’s labels, whereas Updown-trained models performed poorly on Width and Depth.Source: §4.1 · p. 6, Table 6 · p. 8
4.2 Human-like Classification
- Trained on Reddit Human-vs-Rand data, DIALOGRPT showed reasonable zero-shot performance on DailyDialog and PersonaChat.The authors interpret this as reasonable generalization to unseen datasets.Source: §4.2 · p. 6, Table 7 · p. 8
4.3 Ensembling Models
- DIALOGRPT improves candidate ranking compared with perplexity-based methods.The authors report that incorporating human feedback information into response ranking can improve on methods relying solely on relevance measures.Source: §4.3 · p. 8
5 Related Work
Prior dialog-ranking work uses generation probabilities, context–response similarity, or engineered features, while reference-based measures assess response quality offline. Related work also examines contrastive learning and how social-media content features relate to popularity.
5 Related Work
- Dialog response rankers use generation probability or perplexity, mutual information, style intensity, context–candidate similarity, or engineered features.Examples include cohesion, coherence, empathy matching, and retrieval matching.Source: §5 · p. 8
- BLEU, BERTScore, and BLEURT estimate response quality through n-gram overlap, contextualized embeddings, and learned sentence-level similarity, respectively.Reference-based measures are not applicable on-the-fly.Source: §5 · p. 8
- Contrastive learning pulls representations of neighboring samples together and pushes non-neighbors apart, and has also been applied to matched context–response pairs.A contrastive framework was reported to establish a new image-classification state of the art.Source: §5 · pp. 8–9
- Social-media studies model user interactions and upvotes, finding popularity can correlate with quality while varying among similarly rated posts and responding to title features.One study decomposes article popularity into content quality and title appeal.Source: §5 · p. 9
6 Conclusion
The authors trained GPT-2-based feedback-prediction models on Reddit human-feedback pairs and combined them with a human-like scoring model to rank machine-generated responses. The models outperform standard baselines, and human evaluation finds improved human preference with the ranking method.
6 Conclusion
- Human evaluation shows the combined ranking method improves human preference for machine-generated dialog responses.The ranking method ensembles feedback-prediction models with a human-like scoring model.Source: §6 · p. 9
- The authors trained GPT-2-based models on 133M human-feedback pairs and released a large-scale feedback-prediction dataset.The models outperform several standard baselines, while conventional dialog perplexity has little predictive power on Reddit feedback.Source: §6 · p. 9
- Future work could integrate ranking models with generation, such as in beam search or reinforcement learning using the ranking score as a reward.Source: §6 · p. 9