Source-linked AI summary
Touch Your Heart: A Tone-aware Chatbot for Customer Care on Social Media
Tianran Hu, Anbang Xu, Zhe Liu, Quanzeng You, Yufan Guo, Vibha Sinha, Jiebo Luo, Rama Akkiraju
TL;DR
Customer-care chatbots on social media address rising demand, but existing work largely overlooks tones that affect user experience. This paper studies tone effects, incorporates tone information into a deep-learning chatbot trained on over 1.5 million Twitter conversations, and finds responses as appropriate as human agents’ and more empathetic.
Problem
Existing customer-care chatbot work largely overlooks tones as a factor affecting user experience in social-media service.
Method
The paper studies eight customer-care tones and integrates learned tone information into a deep-learning chatbot trained on over 1.5 million real Twitter conversations.
Results
The chatbot generates responses as appropriate as human agents’ and is perceived as more empathetic than human agents.
Takeaways & Limitations
Empathetic and passionate tones are identified as beneficial for customer care, and tone-aware response generation shows application value.
Takeaways & Limitations
The evaluation relies on human judgment, and future field studies are needed to examine end-user perceptions and engagement in practice.
Abstract
from arXiv · showhide
Chatbot has become an important solution to rapidly increasing customer care demands on social media in recent years. However, current work on chatbot for customer care ignores a key to impact user experience - tones. In this work, we create a novel tone-aware chatbot that generates toned responses to user requests on social media. We first conduct a formative research, in which the effects of tones are studied. Significant and various influences of different tones on user experience are uncovered in the study. With the knowledge of effects of tones, we design a deep learning based chatbot that takes tone information into account. We train our system on over 1.5 million real customer care conversations collected from Twitter. The evaluation reveals that our tone-aware chatbot generates as appropriate responses to user requests as human agents. More importantly, our chatbot is perceived to be even more empathetic than human agents.
INTRODUCTION
Social-media customer-care chatbots can address high request volumes, but prior systems largely overlook tones that shape user experience. This work studies tone effects and builds a tone-aware chatbot whose responses are judged appropriate and more empathetic than human agents.
- Research gap: Previous customer-care chatbot work focuses mainly on grammatically correct responses while overlooking other factors affecting user experience, including tones.The proposed system addresses this gap by incorporating tones into response generation.
- Formative study: The formative study identifies eight customer-care tones and finds significant, varied effects, including reduced negative emotion from empathy and increased satisfaction from passion.The identified tones are anxious, frustrated, impolite, passionate, polite, sad, satisfied, and empathetic.
- Evaluation: The system is evaluated with human annotations of appropriateness, helpfulness, and tone intensity against responses from real human agents.Statistical tests compare chatbot and human-agent responses across response quality and perceived tone intensity.
- Findings: Human judgments find the chatbot’s responses as appropriate as human agents’ and more empathetic than their responses.The evaluation also indicates that the chatbot generates proper, tone-aware responses.
RELATED WORK
Prior customer-care research examines response speed, compensation, policies, agent skills, and social-media channels, while related work also documents emotional effects of tones. These strands motivate systematic study of tones in social-media customer care.
- Customer-care quality: Customer-care quality research considers response speed, compensation, company policies, and agents’ conversation skills.Reported effects include compensation on satisfaction, repurchase intention, and word-of-mouth activity.
- Social-media customer care: Social-media customer-care studies report stronger expectations for rapid responses than traditional channels and differing expectations across platforms.The cited work also links social-media brand activity with customer relationships and loyalty.
- Tones in customer care: Prior tone research links conversational tones with emotional impacts, company attitudes, word-of-mouth activity, and customer satisfaction.This literature motivates examining tones as a customer-care factor rather than treating them as incidental wording choices.
Chatbot Systems
The paper situates tone-aware customer-care chatbots within deep-learning conversation systems and builds its evidence from a large Twitter dataset and formative tone annotations.
- Chatbot Systems: Rule-based and retrieval-based chatbots are limited by small-scale data or narrow domains, while seq2seq models support large-scale open-domain generation.Standard seq2seq models, however, do not account for conversational meta-information such as tones.
- Chatbot Systems: The dataset was collected from Twitter conversations involving 62 brands across multiple industries.The collection covered technology, airline, retail, and other brands.
- Chatbot Systems: 3.5 million tweets were collected by tracing reply chains from Twitter’s Developer API between August 1, 2016 and June 1, 2017.The process recovered conversations between brand accounts and users.
- Chatbot Systems: After filtering and cleaning, over 1.5 million conversations remained, with 87.6% containing one user request and one agent response.Conversations were restricted to one user and one agent speaking in alternating turns.
- Chatbot Systems: The formative study reduced 53 candidate tones to eight major tones using factor analysis with Principal Components Analysis.The eight tones were anxious, frustrated, impolite, passionate, polite, sad, satisfied, and empathetic.
- Chatbot Systems: Each utterance in the annotation tasks received ratings from five valid workers, whose average became the final rating.The study used validation questions and restricted workers to native English speakers with prior annotation accuracy.
Effects of Agent Tones
The tone-effects study models changes in user tones as a function of agent tones and finds especially beneficial effects for empathetic and passionate responses.
- Effects of Agent Tones: The study fits eight linear regressions, each predicting the change in one user tone from the eight tones used in the intervening agent utterance.The dependent variable is the difference between adjoining user requests, and the independent variables are agent-tone ratings.
- Effects of Agent Tones: The regression analyses explain changes in passionate and empathetic user tones particularly well, with R2 values of 0.72 and 0.68, respectively.These are the highest reported R2 values among the analyzed user tones.
- Effects of Agent Tones: Empathetic agent tone increases user satisfaction and politeness while reducing anxiety, sadness, and frustration.The reported coefficients are 0.59, 0.23, −0.13, −0.25, and −0.12, respectively.
- Effects of Agent Tones: Passionate agent tone increases user satisfaction, passion, and empathy while reducing sadness.The coefficients are 0.39, 0.26, 0.71, and −0.15, respectively.
- Effects of Agent Tones: Polite agent tone has only a marginally significant, limited positive effect on user satisfaction and does not significantly reduce negative user tones.Its satisfaction coefficient is 0.09 with p < 0.1.
- Effects of Agent Tones: Anxious, frustrated, and sad agent tones are associated with increases in selected negative user tones, while impolite tone has no significant effects.Frustrated tone increases user anxiety, and sad tone increases frustration, impoliteness, and anxiety.
- Effects of Agent Tones: The overall analyses motivate embedding empathetic and passionate tones in the tone-aware chatbot.The paper identifies these tones as beneficial because they improve positive user emotions and reduce negative ones.
Tone Keywords
The paper extracts lexical indicators of empathetic and passionate tones from human customer-care responses and uses them to guide tone-aware generation.
- Tone Keywords: The keyword-extraction procedure collects frequently used uni-grams, bi-grams, and tri-grams and compares responses rated at least 3 on a 4-point tone-intensity scale.Term frequency is normalized by the number of terms in each utterance.
- Tone Keywords: The extracted empathetic keywords mainly express apology or understanding of users’ suffering.Examples include “sorry,” “apologize,” “understand,” “inconvenience,” “frustration,” and “confusion.”
- Tone Keywords: The study identifies 28 keywords for empathetic tone and 19 for passionate tone.All extracted keywords are uni-grams, which the paper attributes to the dataset’s relatively small size.
- Tone Keywords: Passionate keywords increase certainty and trustworthiness or express positive emotion.Examples include “!”, “definitely,” “certainly,” “great,” “awesome,” and “love.”
- Tone Keywords: The tone-aware seq2seq model uses extracted keywords as indicators of tone and learns to generate responses with similar styles.The model is designed to embed empathetic and passionate information into response generation.
TONE-AWARE CHATBOT
Standard seq2seq chatbots learn response patterns from paired sequences but do not incorporate meta-information such as tone. The proposed model adds tone information so responses can be generated in an assigned tone.
- Standard seq2seq models learn matching relations between input and output sequence pairs to generate responses.In customer care, the paired sequences are user requests and agent responses.
- Standard seq2seq models cannot control generated response tones because they do not handle meta-information during learning.
- The proposed tone-aware seq2seq model takes tone information as input and learns distinct expressions associated with different tones.Given an assigned tone, the chatbot generates a response in that tone.
Background on Seq2seq Learning
A standard seq2seq model uses an encoder and decoder to learn mappings between matching sequences. During generation, it encodes the input sequence and produces an output sequence step by step.
- A standard seq2seq model consists of an encoder and decoder, typically implemented with recurrent neural networks such as LSTM or GRU.This work selects LSTM for both components.
- During training, the model learns from matching sequences X and Y, such as a user request and its agent response.The encoder processes X, while the decoder uses the representation of X together with Y.
- During generation, the encoder represents the input sequence and the decoder produces output elements sequentially until a maximum step count or end indicator.The generated sequence W is the model output.
Tone-aware Seq2seq Learning
The tone-aware seq2seq model adds a tone indicator to the decoder during training and generation. This lets the decoder learn tone-specific response patterns and generate responses in a requested tone.
- The model trains on user requests, agent responses, and tone information so it can learn distinct expressions for different tones.An empathetic example pairs a delayed-flight request with a response acknowledging frustration.
- Figure 3 depicts training with a user request, agent response, and tone indicator used to teach the decoder tone-specific expression patterns.
- The model represents tone metadata with a one-dimensional indicator vector t and models empathetic and passionate tones.
- During training, t is concatenated to the encoded request and every decoder input vector, allowing the decoder to track tone-specific differences.The concatenation operation is denoted by ⊕.
- During generation, the requested tone indicator t is concatenated with the encoded request and each generated-word embedding before the next decoding step.Generation stops at the maximum step count or an end indicator.
Implementation
The implementation preprocesses Twitter customer-care conversations into tone-labeled sequence pairs, trains an LSTM-based model, and evaluates responses generated under three tone settings against human responses.
- Multi-turn conversations are converted into matching sequence pairs using the current user request together with preceding utterances.One-round conversations use the user request and agent response directly.
- Tone indicators are assigned from empathetic or passionate keywords in agent responses, with a neutral indicator when neither tone is detected.
- 500 sequence pairs are randomly sampled for evaluation, and each evaluation request generates empathetic, passionate, and neutral responses.Generated responses are compared with responses from real human agents.
- Table 4 compares passionate, empathetic, and neutral chatbot responses with human-agent responses across seven conversations.Five conversations are one-round and two are multi-round; multi-round comparisons use the final round with shared prior context.
- The model uses one-layer LSTM encoder and decoder networks with 512 hidden cells and a 10,000-word vocabulary.Word embeddings have size 256.
- The tone indicator is encoded as -1 for empathetic, 0 for neutral, and +1 for passionate tones.
- Training uses the Adam optimization algorithm with an initial learning rate of 0.001.
EVALUATION
The evaluation compares tone-conditioned chatbot responses with real human-agent responses using human ratings of response quality and perceived tone intensity.
- Figure guide: Figure 4 labels the response conditions as HUM for human, PASS for passionate, EMP for empathetic, and NEU for neutral responses.These labels distinguish the human baseline from the three tone-controlled chatbot conditions.
- Evaluation design: Human annotators rate appropriateness, helpfulness, passionate level, and empathetic level for responses generated in three tones and for real agent responses.The generated tones are passionate, empathetic, and neutral.
- Evaluation design: The study uses survey participants who learn the four rating criteria from definitions and examples before rating responses.Participants are recruited through Crowdflower, and failed gold-question checks lead to response removal.
Results
Human evaluation finds that the tone-aware chatbot matches human agents on appropriateness and tone control, but trails them on helpfulness while producing especially empathetic responses.
- Appropriateness: Generated passionate responses are as appropriate as human responses (p = 0.33), while passionate responses exceed empathetic and neutral responses in appropriateness (p < 0.05).Empathetic responses are marginally more appropriate than neutral responses (p < 0.1).
- Helpfulness: Human responses score higher in helpfulness than empathetic, passionate, and neutral generated responses (p < 0.01 for each comparison).No significant helpfulness difference is reported among the three generated tones.
- Tone control: Passionate responses match human responses in passionate level (p = 0.15), while passionate, empathetic, and neutral responses form significantly ordered tone levels (p < 0.01).The reported ordering is passionate above empathetic above neutral.
- Tone control: Empathetic responses receive the highest empathetic-level ratings, exceeding all other response types (p < 0.01 for all three tests).They are followed by passionate, human, and neutral responses; passionate responses are marginally higher than human responses (p < 0.1).
- Overall findings: Overall, the chatbot produces human-perceivable tones, matches human agents in appropriateness, is less helpful, and is as passionate or more empathetic than human responses.The authors identify passionate tone as potentially more broadly appropriate than empathetic tone, while calling for finer-grained study of tone effects.
- Scope and future work: The evaluation relies on human judgment rather than automatic metrics, and future work is proposed to study tone perceptions and engagement in field settings.The authors state that automatic metrics cannot evaluate whether embedded tones are perceived.
- Model extension: The model uses a deep learning seq2seq framework with an indicator bit that controls output styles and could support brand-specific response styles.The proposed extension would provide brand metadata so the chatbot can learn different styles for different brands.
CONCLUSION
The paper studies customer-care tones, identifies passionate and empathetic tones as beneficial, and integrates tone information into a chatbot that generates appropriate, human-perceivable responses.
- Findings: The study identifies passionate and empathetic tones as beneficial for increasing user experience in customer care.The paper links these tones with reduced negative emotion, increased positive emotion, and increased user satisfaction.
- Approach: The proposed deep learning chatbot integrates tone information and generates toned responses to customer-care requests.The evaluation suggests that its embedded tones are readily perceived by annotators.
- Conclusion: The chatbot generates responses perceived as appropriate as human-agent responses and more empathetic than human-agent responses.The paper identifies finer-grained tone effects, end-user engagement, and brand-aware styles as future directions.