Source-linked AI summary
Gmail Smart Compose: Real-Time Assisted Writing
Mia Xu Chen, Benjamin N Lee, Gagan Bansal, Yuan Cao, Shuyuan Zhang, Justin Lu, Jackie Tsay, Yinan Wang, Andrew M. Dai, Zhifeng Chen, Timothy Sohn, Yonghui Wu
TL;DR
Smart Compose addresses the difficulty of drafting longer e-mails by providing frequent, real-time suggestions during composition. The paper develops a large-scale neural language-model system using contextual inputs and production-serving infrastructure, and reports deployment with extensions for personalization and multilingual support. The authors also identify strict latency constraints and fairness concerns as important boundaries of the system.
Problem
Longer e-mail drafting remains time-consuming despite short, one-time reply suggestions, while real-time assistance must operate at Gmail scale and under strict latency requirements.
Method
Smart Compose uses a large-scale neural language model that predicts from the current e-mail prefix while incorporating contextual information such as subject, previous body, date, time, and locale.
Results
Smart Compose is served in Gmail as a real-time system providing context-dependent and diverse suggestions, with reported experimental comparisons across model and serving architectures.
Takeaways & Limitations
The system extends Gmail writing assistance with personalization and multilingual support while addressing model design, evaluation, serving, privacy, and fairness concerns.
Takeaways & Limitations
Most advanced training techniques and architectures improved suggestion quality but failed to meet Smart Compose's strict latency constraints.
Abstract
from arXiv · showhide
In this paper, we present Smart Compose, a novel system for generating interactive, real-time suggestions in Gmail that assists users in writing mails by reducing repetitive typing. In the design and deployment of such a large-scale and complicated system, we faced several challenges including model selection, performance evaluation, serving and other practical issues. At the core of Smart Compose is a large-scale neural language model. We leveraged state-of-the-art machine learning techniques for language model training which enabled high-quality suggestion prediction, and constructed novel serving infrastructure for high-throughput and real-time inference. Experimental results show the effectiveness of our proposed system design and deployment approach. This system is currently being served in Gmail.
1 INTRODUCTION
Smart Compose extends Gmail assistance from brief, one-time replies to real-time, context-dependent suggestions throughout message composition. Its deployment must balance writing support with strict latency, scale, evaluation, and serving requirements.
- 1 INTRODUCTION: Longer drafts remain time-consuming despite initial assistance, creating a need for accurate and frequent suggestions during composition.
- 1 INTRODUCTION: Smart Compose provides real-time, interactive, context-dependent suggestions while users compose new Gmail messages.It reduces repetitive idiomatic writing and supports composing from scratch rather than only generating short replies.
- 1 INTRODUCTION: A neural language model trained on large-scale e-mail data makes instant predictions as users type.
- 1.1 Challenges: The paper addresses model evaluation, large-scale inference, serving, personalization, internationalization, privacy, and fairness as deployment challenges.
- 1.1 Challenges: The system requires 90th-percentile end-to-end latency below 60ms while serving suggestions on almost every keystroke.
- 1.1 Challenges: Supporting more than 1.5 billion diverse Gmail users requires sufficient model capacity for tailored, high-quality suggestions across subtly different contexts.
2 RELATED WORK
Smart Compose builds on language-modeling research and differs from Smart Reply in serving new-message composition with repeated, real-time suggestions. Its broader system extends language-model applications with personalization and multilingual support.
- 2 RELATED WORK: Smart Compose applies large-scale neural language modeling to generate suggestions for subsequent words and accelerate user typing.
- 2 RELATED WORK: Unlike Smart Reply, Smart Compose suggests extensions for every sentence prefix during real-time composition rather than generating one response per mail.
- 2 RELATED WORK: Personalization is treated as a language-model adaptation technique, with Smart Compose using an architecture related to interpolated n-gram and recurrent neural models.
- 2 RELATED WORK: Multilingual modeling is presented as an extension relevant to internationalization and an active NLP research area.
3 FINDING THE RIGHT MODEL
Smart Compose treats suggestion generation as conditional sequence prediction, compares language-model and Seq2Seq architectures, and evaluates quality under production-oriented constraints. Context features improve model quality, while model choice balances prediction quality against serving latency and suggestion coverage.
- Task and inputs: Smart Compose predicts variable-length token sequences from the typed prefix and contextual information such as subject, previous mail, date, time, and locale.These conditional inputs are incorporated through model embeddings or context encoders.
- Model comparison: The study compares recurrent and Transformer language models, multiple model sizes, and state-of-the-art Seq2Seq architectures for Smart Compose.The Seq2Seq experiments include Transformer Big and RNMT+ models alongside an LSTM Seq2Seq baseline.
- Model quality: 0.13 lower test log perplexity results when subject and previous-mail embeddings are added to LSTM-2-1024, showing the value of contextual information.At similar parameter counts, an 84M-parameter Transformer also has 0.18 lower log perplexity than an 80M-parameter LSTM.
- Model quality: Higher model capacity lowers perplexity, and Transformers generally achieve better quality than LSTMs at similar parameter counts.The comparison is reported for models using the LM-A approach.
- Evaluation: Perplexity differences generally carry over to ExactMatch, especially for longer suggestions, but overall average gaps are smaller because most suggestions are short.ExactMatch is averaged across suggestion lengths up to 15 after equalizing model coverage with confidence thresholds.
- Model comparison: LM-A and LM-B produce very similar model quality and convergence time when model type and capacity are held constant.Inference uses beam search, while confidence thresholds determine which top suggestions are shown.
- Model comparison: Seq2Seq LSTM outperforms similarly sized LSTM language models, indicating that attention-based encoder-decoder modeling is more effective for contextual information.The study also evaluates larger Transformer and RNMT+ Seq2Seq models on the task.
4 PRODUCTION SYSTEM
Smart Compose uses streaming, incremental language-model inference and beam search to generate suggestions as users type. Production model selection balances suggestion quality against strict latency and scale requirements.
- Serving and inference: Each request alternates prefix encoding and beam-search steps, with prefix encoding producing hidden states and beam search exploring plausible suggestion extensions.Partial-word completions and application-controlled stopping rules constrain beam search.
- Serving and inference: Cached hidden-state results make prefix-encoding cost typically proportional to the text typed since the previous request.Sticky server assignment enables reuse across requests.
- Latency and model choice: Beam-search latency increases with model size, while Transformer decoding is slower than comparable LSTM decoding and becomes increasingly costly for longer suggestions.Self-attention requires retaining keys and values from previous decoding steps across model layers.
- Latency and model choice: LM-A is the most production-appropriate model because Smart Compose faces strict latency constraints and very high request volume.ExactMatch is more relevant to production than log perplexity, where the quality gap between RNN and Transformer models is less evident.
- Contextual suggestions: Production examples show that the model can capture subject and previous-email context despite using averaged embeddings for each field.The production model's suggestions are documented in Table 6.
- Serving infrastructure: Hardware acceleration and cross-request batching support the throughput required for production inference.Cloud TPU hardware and XLA optimization were used to accelerate inference, while batching combines prefix-encoding and beam-search work.
5 PERSONALIZATION
Personalization adapts Smart Compose to individual writing styles with lightweight per-user language models interpolated with the global model. Offline and production results support a moderate interpolation weight rather than relying exclusively on either model.
- Motivation and model: A lightweight language model is trained for each user to capture personal writing style and vocabulary without adding too much serving burden.The personal model uses a user's recent sent emails and a constrained personal vocabulary.
- Motivation and model: Final prediction probabilities linearly interpolate personal and global model probabilities at each beam-search time step.The interpolation weight α is kept constant to limit personalization overhead.
- Evaluation: The blending-weight evaluation compared 11 α values on real user data while adjusting confidence thresholds to keep coverage roughly comparable with the global-only model.Global-only corresponds to α = 0.
- Evaluation: Personalized models outperform the global model on ExactMatch when α is properly set, with performance peaking at α = 0.4.Performance declines when α is increased further.
- Evaluation: Production personalization produced around 6% relative CTR gain and 10% relative ExactMatch gain.The authors attribute the qualitative improvement to more tailor-made suggestions for individual users.
6 MULTILINGUAL MODEL
The multilingual extension compares word and wordpiece models across Spanish, French, Italian, and Portuguese. A multilingual wordpiece model was deployed because the overall quality and maintenance trade-off supported production use, despite language-specific quality differences.
- Model comparison: The study evaluates multilingual wordpiece models against monolingual word and monolingual wordpiece models for four languages using matched vocabulary, architecture, and training settings.The dataset contains approximately 4 billion Spanish, 2 billion French, 1 billion Italian, and 2 billion Portuguese messages.
- Model comparison: Monolingual word models achieve the lowest perplexities across all four languages, while monolingual wordpiece models generally outperform multilingual wordpiece models on ExactMatch.Spanish shows the clearest quality gaps among the compared approaches.
- Model comparison: For Portuguese, the multilingual wordpiece model outperforms both monolingual models, benefiting from data in similar languages.This result differs from the pattern observed for Spanish, French, and Italian.
- Production deployment: A multilingual wordpiece model was deployed for all four languages based on overall quality and maintenance requirements.Language-specific triggering confidence thresholds were used to obtain similar coverage across languages and with English.
7 FAIRNESS AND PRIVACY
The paper addresses fairness and privacy risks arising from Smart Compose’s language-model training and deployment. It tests for bias and memorization, removing gendered suggestions and checking that only common phrases are memorized.
- Fairness: Gender–occupation bias appeared in suggestions, such as “meet him” after “investor” and “meet her” after “nurse.”The authors removed suggestions containing gender pronouns to limit this bias.
- Fairness: Smart Compose models trained on natural-language data can reflect human cognitive biases.This motivates explicit testing of model suggestions for bias.
- Privacy: The authors tested memorization to ensure the model retained only common phrases used by multiple users.This testing was motivated by the model’s training on billions of phrases and sentences and findings from Carlini et al.
8 FUTURE WORK
Future work targets better suggestion quality without violating Smart Compose’s strict latency requirements. The paper also proposes hardware and inference optimizations and explores variational language-modeling methods.
- Future directions: Advanced architectures and training techniques improved suggestion quality but generally failed to meet strict latency constraints.The authors therefore seek quality improvements with minimal latency cost.
- Future directions: The authors plan to optimize inference and serving while exploiting newer hardware accelerators.This direction is intended to reduce the latency cost of higher-quality models.
- Future directions: A locally windowed Transformer is proposed as a way to preserve quality improvements while limiting self-attention during decoding.Self-attention would operate over a fixed-size window rather than all previous decoding steps.
- Future directions: Variational auto-encoders are identified as a possible route to efficient learning and inference in latent-variable language models.The paper notes that VAE extensions to recurrent neural-network language models could be applied to RNN-based language models.
9 CONCLUSION
Smart Compose provides real-time, context-dependent, and diverse writing suggestions in Gmail while addressing the practical challenges of deploying large-scale language models. The paper also extends the system with personalization and multilingual support and compares model and serving architectures experimentally.
- Conclusion: Smart Compose improves Gmail writing by providing real-time, context-dependent, and diverse suggestions as users type.The system is presented as a large-scale language-model application for assisted composition.
- Conclusion: The paper details challenges involving model design, evaluation, serving, privacy, and fairness.These challenges arise in developing and deploying the system at scale.
- Conclusion: Personalization and multilingual support extend the main Smart Compose system.The conclusion identifies both as additional system features.
- Conclusion: Experiments compare performance across different model and serving architectures.The paper reports extensive experimental results for these comparisons.