Source-linked AI summary
Smart Reply: Automated Response Suggestion for Email
Anjuli Kannan, Karol Kurach, Sujith Ravi, Tobias Kaufmann, Andrew Tomkins, Balint Miklos, Greg Corrado, Laszlo Lukacs, Marina Ganea, Peter Young, Vivek Ramavajjala
TL;DR
Smart Reply tackles the challenge of helping users compose short email replies, especially on mobile, by generating complete response suggestions. It combines LSTM response prediction with scalable response-space generation, semantic diversity, and triggering, and is used for 10% of Inbox mobile replies. The paper also addresses production constraints including throughput, privacy, response quality, and utility.
Problem
Composing short replies is time-consuming on mobile, while deploying automated suggestions globally requires handling response quality, utility, scalability, and privacy.
Method
Smart Reply combines LSTM sequence-to-sequence response prediction with approximate search, semi-supervised response-space generation, semantic-diversity selection, and a triggering model.
Results
10% of mobile replies in Inbox are composed with assistance from Smart Reply.
Takeaways & Limitations
Smart Reply provides a deployed end-to-end system for generating short, complete email responses at production scale.
Takeaways & Limitations
Privacy protections restrict inspection to aggregated statistics on anonymized, frequent content, making model-quality verification and debugging more complex.
Abstract
from arXiv · showhide
In this paper we propose and investigate a novel end-to-end method for automatically generating short email responses, called Smart Reply. It generates semantically diverse suggestions that can be used as complete email responses with just one tap on mobile. The system is currently used in Inbox by Gmail and is responsible for assisting with 10% of all mobile responses. It is designed to work at very high throughput and process hundreds of millions of messages daily. The system exploits state-of-the-art, large-scale deep learning. We describe the architecture of the system as well as the challenges that we faced while building it, like response diversity and scalability. We also introduce a new method for semantic clustering of user-generated content that requires only a modest amount of explicitly labeled data.
1. INTRODUCTION
Smart Reply addresses the difficulty of composing short mobile email replies with an end-to-end system that predicts, filters, and presents diverse response suggestions. It was deployed in Inbox and assists with 10% of mobile email replies.
- Motivation: 25% of replies in an initial study contained 20 tokens or fewer, motivating one-tap suggestions for brief messages.The study covered several million email-reply pairs.
- System: Smart Reply uses sequence-to-sequence learning with LSTMs to predict responses from incoming messages.The framework produces a generative distribution over possible replies.
- System: The system combines approximate LSTM response selection, offline response-set generation, semantic-intent diversity, and a triggering model.The triggering model avoids showing suggestions when short replies are inappropriate or unnecessary.
- Contributions: Response selection uses deep learning trained on billions of words, while response-set generation uses semi-supervised semantic understanding of user-generated content.The combined approach is presented as a novel end-to-end method for short, complete email responses.
- Deployment: 10% of Inbox mobile email replies are assisted by Smart Reply after its deployment.The system was designed to address response quality, utility, scalability, and privacy challenges.
2. RELATED WORK
Smart Reply extends response prediction and semantic response-space discovery into a single production-oriented system for email. Its novelty lies in addressing quality, utility, scalability, and privacy together rather than treating email as a merely on-topic dialogue domain.
- Scope: Existing related work covered response prediction and target response-space identification, but email response suggestion lacked established literature or benchmarks.The paper frames automated email response suggestion as a distinct task rather than a standard machine-learning problem.
- Response prediction: The paper uses sequence-to-sequence LSTM modeling for full response prediction instead of phrase-based statistical machine translation.Sequence-to-sequence learning had also been applied to image captioning and speech recognition.
- Production setting: Its production deployment introduced response-quality, utility, scalability, and privacy challenges not considered in the related works.These challenges motivated the paper’s novel solutions.
- Response prediction: Unlike social-media and movie-dialogue systems, email responses frequently need to address an expressed request or intent.Merely related or on-topic responses can be acceptable in those other domains but are less aligned with email’s stated needs.
- Response-space identification: Expander is used to discover semantic-intent clusters from user-generated content, a use the paper identifies as novel.The approach is selected because it scales to large email data and large output spaces.
- Contribution: Smart Reply is presented as the first system to address these tasks together in a single end-to-end, deployable system.This combines response prediction with target response-space discovery.
3. SELECTING RESPONSES
Smart Reply models response likelihood with an LSTM and searches a curated response space efficiently, addressing quality and scalability constraints for production deployment.
- Response objective: The system seeks the most likely response r∗ for an incoming message o from the possible-response space R.It scores candidate responses and selects the highest-scoring one.
- LSTM model: The sequence-to-sequence LSTM encodes incoming-message tokens and predicts conditional probabilities for response tokens.The response probability is factorized into next-token probabilities conditioned on the message and preceding response tokens.
- Inference: Beam search approximates response decoding by retaining the top b response prefixes at each timestep.The method supports sampling, approximate maximum-likelihood decoding, and scoring specific candidates.
- Production challenges: The response space contains only high-quality candidates, while normalization penalizes broadly applicable replies and diversity methods address redundant outputs.These choices target quality, specificity, and utility rather than unconstrained generation.
- Scalability: O(bl) trie-constrained beam search avoids exhaustive O(|R|l) scoring and matches exhaustive search’s top response 99% of the time at b = 128.The search complexity depends on beam size and maximum response length rather than the size of R.
- Production challenges: Privacy protections encrypt email data and restrict inspection to aggregate statistics, making model verification and debugging more complex.Only frequent words are retained, further limiting inspection.
4. RESPONSE SET GENERATION
Smart Reply constructs a structured response space by clustering frequent messages according to semantic intent using a graph-based semi-supervised method with limited manual seeds.
- Goal: The target response space must capture both linguistic variability and response intents for scoring and suggestion diversity.It is used downstream to select responses and promote diversity among them.
- Semantic representation: Frequent response messages are canonicalized and partitioned into semantic clusters whose members share a communication purpose despite differing wording.Examples include “thank you,” “sorry,” “cannot make it,” and humorous responses.
- Graph construction: A graph links response and manually labeled nodes through lexical feature nodes, with optional edges representing inter-message relations.The graph is built from frequent responses, ngrams, skip-grams, and labeled examples.
- Semi-supervised learning: Semi-supervised label propagation optimizes jointly over graph nodes using seed labels, neighborhood consistency, and a prior-distribution penalty.The method uses EXPANDER and typically starts with 100 clusters and 3–5 labeled seed examples per cluster.
- Semi-supervised learning: EXPANDER outputs semantic-label distributions for every node, after which the highest-scoring labels are assigned and low-scoring labels filtered.Figure 4 illustrates this semantic clustering process.
- Validation and expansion: Repeated propagation expands cluster membership and discovers up to 5X new interpretable clusters, whose top members are human-validated.The validated response-intent set supports approximate response search and diversity enforcement.
5. SUGGESTION DIVERSITY
Suggestion diversity prevents Smart Reply from showing redundant responses and deliberately preserves contrasting response types when the highest-scoring candidates are skewed.
- Motivation: Simply presenting the top-scoring responses can produce highly similar suggestions, reducing the chance that at least one is useful.The system therefore treats response quality and utility jointly.
- Intent structure: Each response is assigned one validated intent, enabling diversity selection by semantic communication purpose.An intent may represent confirming, asking for time, or rejecting participation.
- Intent diversity: The diversity algorithm iterates through responses by decreasing score and keeps only the highest-scored representative of each intent.Redundant responses are omitted from the final suggestion list.
- Polarity diversity: When the top candidates are positive and lack negatives, the third suggestion is replaced with a negative response.A second LSTM pass searches only negative responses to find that replacement.
- Polarity diversity: If an incoming message produces exclusively negative responses, the system applies an analogous policy to enforce a positive suggestion.The resulting top-scoring responses are shown to the user.
6. TRIGGERING
A triggering module filters emails unlikely to benefit from short replies before invoking the more expensive LSTM, enabling fast processing at production scale.
- Filtering: The trigger excludes messages where short replies are inappropriate or unnecessary, including open-ended questions, sensitive topics, promotions, and automated updates.If the decision is negative, no suggestions are shown.
- Impact: Smart Reply is produced for roughly 11% of messages, reducing useless suggestions and calls to expensive LSTM inference.The latter reduces infrastructure cost.
- Requirements: The trigger must distinguish messages where a response is not expected from messages that merely admit valid candidate replies.This differs from simply scoring a set of responses.
- Requirements: Because the system processes hundreds of millions of messages daily, the trigger aims to process each message within milliseconds.Fast filtering is required before running downstream inference.
- Trigger model: A feedforward neural network assigns each incoming message a probability score and invokes LSTM scoring when the score exceeds a threshold.Its training labels indicate whether the message received a mobile response, modeling P(y = true | o).
7. EVALUATION AND RESULTS
The evaluation measures Smart Reply’s end-to-end usage, component quality, response diversity, and scalability. Results show substantial mobile use, improved response ranking, broad suggestion coverage, and a measurable benefit from semantic diversity.
- Overall results: 10% of all mobile replies use Smart Reply, the system’s principal end-to-end usage metric.The evaluation combines offline component analyses with online experiments on a subset of accounts.
- Triggering model: The triggering model achieves an AUC of 0.854, while triggered messages comprise 11% of the deployed system.The model is evaluated using temporally ordered training and test data with precision, recall, and ROC-AUC metrics.
- Response selection: 17.0 perplexity for Smart Reply compares with 31.4 for a Katz-backoff 5-gram model on the same data.Both values are computed using response terms only.
- Response selection: Smart Reply’s LSTM significantly outperforms both Frequency and Multiclass-BOW baselines on response ranking.Conditioning on the original message improves ranking over frequency, while the sequence-to-sequence framework also exploits examples whose responses are absent from the response set.
- Diversity: 12.9k daily unique suggestions span 376 semantic clusters, with users selecting 31.9% of suggestions and 83.2% of clusters.These statistics motivate a large response space and semantic diversity rather than a small set of canned replies.
- Diversity: Disabling diversity decreases click-through rate by roughly 7.5% relative, while users select 45%, 35%, and 20% of suggestions from positions one, two, and three.The third position is typically used for diverse responses, linking engagement loss to the diversity component.
8. CONCLUSIONS
Smart Reply is an end-to-end system for generating short email responses that addresses response quality, diversity, utility, and scalability challenges. In Inbox by Gmail, 10% of mobile replies are composed with its assistance.
- Smart Reply generates short, complete email responses with a deep LSTM model given an incoming email message.
- Response quality is supported by a carefully constructed response space whose messages are identified through semantic clustering.
- The system increases suggestion utility by enforcing diversity among suggestions and filtering traffic where suggestions would not be useful.
- 10% of mobile replies in Inbox are composed with assistance from Smart Reply.