PaperlayerView exact revision on arXiv

Source-linked AI summary

Interpretable User Satisfaction Estimation for Conversational Systems with Large Language Models

Ying-Chun Lin, Jennifer Neville, Jack W. Stokes, Longqi Yang, Tara Safavi, Mengting Wan, Scott Counts, Siddharth Suri, Reid Andersen, Xiaofeng Xu, Deepak Gupta, Sujay Kumar Jauhar, Xia Song, Georg Buscher, Saurabh Tiwary, Brent Hecht, Jaime Teevan

arXiv:2403.12388v2cs.IRcs.AI

TL;DR

User satisfaction estimation needs to capture why users are satisfied or dissatisfied, while existing approaches can be opaque or limited in generalizing across conversational patterns. SPUR uses supervised iterative prompting to learn rubrics and score conversations. It outperforms existing methods across conversational systems and makes satisfaction dimensions interpretable through observed user behavior.

  • Problem

    Existing approaches can be opaque or insufficiently generalizable for understanding diverse conversational patterns that signal user satisfaction.

    Source: §1 · p. 1

  • Method

    SPUR iteratively uses labeled examples to extract satisfaction signals, summarize them into rubrics, and apply the rubrics to score unseen conversations.

    Source: §1 · p. 2

  • Results

    SPUR outperforms existing methods across different conversational systems, and its rubrics ground satisfaction dimensions in observed user behavior.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Rubrics provide interpretable satisfaction dimensions and can add signals to embedding-based models for performance improvement on user satisfaction estimation.

    Source: §5 · p. 9

    Estimating satisfaction across millions of conversations is costly, and the robustness of distilling knowledge from GPT-4 needs further study.

    Source: §5 · p. 9

Abstract

from arXiv · show

Accurate and interpretable user satisfaction estimation (USE) is critical for understanding, evaluating, and continuously improving conversational systems. Users express their satisfaction or dissatisfaction with diverse conversational patterns in both general-purpose (ChatGPT and Bing Copilot) and task-oriented (customer service chatbot) conversational systems. Existing approaches based on featurized ML models or text embeddings fall short in extracting generalizable patterns and are hard to interpret. In this work, we show that LLMs can extract interpretable signals of user satisfaction from their natural language utterances more effectively than embedding-based approaches. Moreover, an LLM can be tailored for USE via an iterative prompting framework using supervision from labeled examples. The resulting method, Supervised Prompting for User satisfaction Rubrics (SPUR), not only has higher accuracy but is more interpretable as it scores user satisfaction via learned rubrics with a detailed breakdown.

1 Introduction

The introduction motivates user satisfaction estimation that explains conversational patterns, not only satisfaction labels. It presents SPUR as a supervised prompting framework that learns interpretable rubrics and reports stronger performance across conversational systems, including ways to scale rubric use.

  • Existing approaches prioritize classification accuracy or rely on opaque embeddings, while domain-specific features may not generalize across varied conversational patterns.Source: §1 · pp. 1–2
  • SPUR uses iterative prompting to extract satisfaction signals from labeled examples, summarize them as rubrics, and score unseen conversations.Source: §1 · p. 2
  • SPUR learns interpretable satisfaction and dissatisfaction rubrics for different conversational systems, with pattern distributions linked to overall satisfaction and differing across domains.Source: §1 · p. 2
  • Rubric items can be distilled into embedding-based models for application without LLM prompting, and adding them as features improves embedding-only models when more training data is available.Source: §1 · p. 2
  • SPUR outperforms existing methods across different conversational systems when training data is limited.Source: §1 · p. 2

2 Problem Definition and Related Work

The paper defines user satisfaction estimation as predicting and explaining a conversation’s overall feedback label through interpretable satisfaction and dissatisfaction patterns. It situates this goal amid prior sentiment, content-analysis, embedding, and LLM-prompting approaches and their stated limitations.

  • A conversation comprises multiple user-agent turns and has an overall satisfaction label supplied by thumb feedback, such as like or dislike.Source: §2 · p. 2
  • The goal is to predict satisfaction labels for unseen conversations and explain predictions using a subset of latent SAT/DSAT pattern classes.Source: §2 · p. 2
  • Prior USE work includes sentiment analysis and human-annotated content analysis, but sentiment distribution is not equivalent to satisfaction estimation (Song et al., 2023).Source: §2 · p. 3
  • Embedding-based approaches model intents, dialogue-act sequences, sentiment, or satisfaction dynamics, while zero-shot instructions and few-shot examples may not fit or cover actual conversational patterns.Source: §2 · p. 3

3 SPUR

SPUR uses a three-phase prompting framework to estimate user satisfaction from a small set of labeled conversations. It extracts satisfaction patterns, summarizes them into rubrics, then uses those rubrics to score unlabeled conversations.

3 SPUR

  • SPUR combines supervised extraction, rubric summarization, and rubric-based estimation to improve accuracy, generalization, and interpretability.It learns satisfaction and dissatisfaction rubrics from labeled conversations, then applies them to score unlabeled conversations.Source: §3 · p. 3

3.1 Supervised Extraction

  • For each labeled conversation, GPT-4 extracts up to three candidate satisfaction or dissatisfaction patterns relevant to its label.The extracted patterns are collected across the training set for rubric learning.Source: §3.1 · pp. 3–4

3.2 Rubric Summarization

  • Rubric Summarization condenses variable, semantically similar extracted patterns into recurring satisfaction and dissatisfaction pattern classes.An iterative minibatch process updates each rubric, incorporating the previous rubric into the next batch; the experiments use ten items per rubric.Source: §3.2 · p. 4

3.3 User Satisfaction Estimation

  • For each rubric item, GPT-4 marks whether the behavior appears and scores its likely satisfaction impact, then subtracts dissatisfaction scores from satisfaction scores.A “Yes” receives a 1–10 impact score, while a “No” receives 0.Source: §3.3 · p. 5

4 Evaluation

The evaluation compares SPUR with LLM and embedding-based baselines on one general-purpose and three task-oriented conversational datasets. The task-oriented datasets contain fewer conversations, and their turn-level satisfaction labels are processed into conversation-level labels.

4 Evaluation

  • SPUR is evaluated quantitatively against embedding-based approaches and ablated LLM methods.The comparison includes GPT-4 zero-shot and few-shot prompting, manually selected features, and embedding-based baselines.Source: §4 · p. 5, §4.1 · p. 5

4.1 Baselines

  • The baselines comprise Lin-ada, a linear-regression model with ada-002 embeddings; USDA, which jointly optimizes satisfaction and sequential dialogue-act dynamics; ASAP; and GPT-4 zero-shot and few-shot methods.ASAP models user satisfaction across turns via a Hawkes Process.Source: §4.1 · p. 5

4.2 Dataset

  • Evaluation uses 50K anonymized Bing Copilot conversations and three smaller task-oriented datasets: MWOZ (1,155), SGD (1,638), and ReDial (1,387).The task-oriented datasets’ turn-level satisfaction labels are processed into whole-conversation labels.Source: §4.2 · p. 5

4.3 USE under Few-Shot Setting.

Under few-shot conditions, SPUR and zero-shot prompting outperform embedding methods, while learned rubrics improve on manually selected features and few-shot prompting. Dataset-specific rubrics gain an average 13% in weighted F1 over Bing Copilot rubrics, and rubric scores vary with thumb feedback.

4.3 USE under Few-Shot Setting.

  • Embedding methods perform worse than SPUR, while ZeroShot and SPUR identify satisfaction patterns effectively from limited data.The authors attribute embedding methods’ lower weighted F1 to difficulty generalizing with small training sets, particularly under class imbalance.Source: §4.3 · p. 6, Table 1 · p. 7

4.4 Importance of Rubric Summarization.

  • 13% average weighted F1 gain follows replacing Bing Copilot rubrics with dataset-specific rubrics on the task-oriented datasets.The reported rubric-set differences indicate that the target datasets learn different satisfaction and dissatisfaction items from Bing Copilot.Source: §4.4 · p. 6, Table 2 · p. 7

4.5 Rubric vs. Thumb Feedback.

  • Satisfaction rubric items score higher for Like feedback, while dissatisfaction items score higher for Dislike; all twenty items differ significantly by feedback.The figure marks items significantly correlated with thumb feedback; the authors report p < 0.05.Source: §4.5 · p. 7, Figure 3 · p. 7

4.6 Pattern Variance for Different Conversational Systems.

Satisfaction and dissatisfaction patterns differ with conversational-system goals, while rubric-based signals can be distilled or combined with embeddings to extend their use beyond prompting. The reported comparisons show distinct performance patterns across rubric distillation, production feedback, and training-set sizes.

4.6 Pattern Variance for Different Conversational Systems.

  • Figure 4 contrasts general conversational rubric items learned from Bing Copilot with booking-specific patterns learned from MWOZ.Its top row shows Bing Copilot items and its bottom row MWOZ items; bars indicate how often each item appears in conversations.Source: §4.6 · p. 7, Figure 4 · p. 8
  • Bing Copilot dissatisfaction more often reflects inaccurate information, whereas MWOZ dissatisfaction more often reflects low proactivity or compromised preferences.Bing Copilot is a general-purpose question-answering system; MWOZ is a booking-reservation system.Source: §4.6 · p. 8

4.7 Knowledge Distillation.

  • Distilled classifiers for Gratitude and Negative Feedback compare ada-002 and E5 embeddings with XLM-RoBERTa, and ada-002 is most effective for distillation.The evaluation uses AUC, with higher AUC indicating more successful distillation of GPT-4 knowledge.Source: §4.7 · p. 8, Figure 5 · p. 8
  • In one production week, users gave more positive feedback through thumbs and more negative feedback through utterances than through thumbs.Figure 6 reports textual and thumb feedback frequencies relative to observed thumb feedback.Source: §4.7 · p. 8, Figure 6 · p. 9

4.8 Rubrics as Features

  • SPUR gives the best F1 results at smaller training sizes, while SPUR-Lin-ada improves as training data grows relative to SPUR and ASAP.Figure 7 compares training sizes from 400 to 90K; SPUR was not retrained above 10K, so its larger-size line uses the 10K-trained model.Source: Figure 7 · p. 9, §4.8 · p. 9
  • Adding SPUR rubric metrics to embedding features provides additional user-satisfaction signals not captured by conversation embeddings.For the larger training sizes, SPUR-Lin-ada combines rubric scores with OpenAI ada-002 embeddings.Source: §4.8 · p. 9, Figure 7 · p. 9

5 Conclusion and Limitations

The conclusion presents SPUR as an accurate, interpretable approach across conversational-system types and describes uses of its learned rubrics. It also notes that cost constrains large-scale application and that the robustness of the distillation approach needs further study.

Conclusion

  • SPUR outperforms existing methods across conversational-system types and provides interpretable rubric scores grounded in observed user behavior.The conclusion also reports insights into factors influencing satisfaction and rubric utility for knowledge distillation and coverage analysis.Source: §5 · p. 9
  • Combining rubric-item scores with conversational embeddings supplies extra signals for improving satisfaction estimation across training-set sizes.The conclusion reports this combination as useful for performance improvement on user satisfaction estimation.Source: §5 · p. 9

Limitations

  • SPUR remains costly for estimating satisfaction across millions of conversations, and the robustness of its proposed knowledge distillation requires thorough study.The authors identify reducing large-scale cost as future work.Source: §5 · p. 9

A Prompts

The appendix prompts define how SPUR extracts satisfaction explanations, summarizes them into rubrics, and applies those rubrics to estimate satisfaction and dissatisfaction.

A.1 Supervised Extraction Prompt

  • The supervised extraction prompt asks for concise, conversation-grounded reasons why a user is satisfied, capped at three mutually exclusive reasons.It instructs the model to return NONE when no user utterance expresses satisfaction and not to mention the like signal.Source: §A.1 · pp. 11–12
  • The extraction prompt uses the conversation history as input and requests the main reasons for user satisfaction as output.The prompt places user-agent utterances between conversation tags.Source: §A.1 · p. 12

A.2 Rubric Summarization Prompt

  • Rubric summarization turns example satisfaction explanations into a concise, user-centric list of mutually exclusive rubric items.The output is a numbered list capped by the requested number of items.Source: §A.2 · p. 12
  • The rubric prompt supplies prior examples, including earlier rubric items, as material to summarize.The prompt's example field includes the prior rubric set and satisfaction explanations.Source: §A.2 · p. 12

A.3 User Satisfaction Estimation Prompt

  • The estimation prompt applies both satisfaction and dissatisfaction rubrics to the conversation history.Each rubric contains ten criteria; the model answers Yes or No and scores applicable criteria from 1 to 10, with 0 for No.Source: §A.3 · p. 12
  • For each rubric question, the model evaluates user utterances in context, returns one confident Yes-or-No answer, and scores every item.The prompt says not to use agent responses except to contextualize the user's response, and requires answers to all ten questions.Source: §A.3 · pp. 12–13

B Labeling Adjustment for the Open Data

The appendix converts turn-level labels in open datasets into conversation-level labels, using the first non-neutral label when both satisfaction and dissatisfaction occur.

B Labeling Adjustment for the Open Data

C Experiment Setup

Experiments use GPT-4 to train and evaluate SPUR, with SPUR-Lin-ada run on an NVIDIA A100. Each experiment runs once with 80% of the data used for testing.

  • GPT-4 is used for the entire SPUR training and evaluation process.Source: C Experiment Setup · p. 13
  • Each experiment runs once, with 80% of the data reserved for testing; SPUR-Lin-ada runs on an NVIDIA A100.Source: C Experiment Setup · p. 13
  • The experiments use three top-k SAT/DSAT patterns per conversation, minibatches of 100 patterns, and 10 items in each satisfaction rubric.Source: C Experiment Setup · p. 13

D User Satisfaction Model

Adding SPUR rubrics to conversation embeddings improves both baseline models, with the best regression layer depending on training-set size. The model combines rubric scores and embeddings to predict a user satisfaction score.

E Usage of AI Assistants

The section presents satisfaction and dissatisfaction patterns expressed through user behavior and reactions to assistant responses, including interaction, usefulness, and task-completion cues.

Related papers

Loading 2403.12388v2…