Source-linked AI summary
Comparing Speech and Keyboard Text Entry for Short Messages in Two Languages on Touchscreen Phones
Sherry Ruan, Jacob O. Wobbrock, Kenny Liou, Andrew Ng, James Landay
TL;DR
Modern speech recognition was not well characterized relative to improved smartphone keyboards. The paper compares speech and keyboard transcription in English and Mandarin under controlled laboratory conditions, finding speech nearly three times faster, with fewer corrected entry errors but slightly more final errors. These results define upper-bound performance and motivate evaluation in realistic settings.
Problem
Little was known about how modern speech recognizers perform relative to improved mobile keyboards, even under ideal conditions.
Method
The study compared Baidu Deep Speech 2 with English QWERTY and Mandarin Pinyin keyboard entry on an iPhone 6 Plus in a controlled laboratory transcription experiment.
Results
Speech was nearly three times faster than the keyboard and made fewer errors during entry, but left slightly more errors after entry was complete.
Takeaways & Limitations
The results provide an upper-bound comparison for modern mobile speech and keyboard entry and support further investigation of speech input.
Takeaways & Limitations
The results were obtained under ideal laboratory conditions and with short-message transcription, so performance in noise, motion, distraction, or broader text-entry tasks requires further study.
Abstract
from arXiv · showhide
With the ubiquity of mobile touchscreen devices like smartphones, two widely used text entry methods have emerged: small touch-based keyboards and speech recognition. Although speech recognition has been available on desktop computers for years, it has continued to improve at a rapid pace, and it is currently unknown how today's modern speech recognizers compare to state-of-the-art mobile touch keyboards, which also have improved considerably since their inception. To discover both methods' "upper-bound performance," we evaluated them in English and Mandarin Chinese on an Apple iPhone 6 Plus in a laboratory setting. Our experiment was carried out using Baidu's Deep Speech 2, a deep learning-based speech recognition system, and the built-in Qwerty (English) or Pinyin (Mandarin) Apple iOS keyboards. We found that with speech recognition, the English input rate was 2.93 times faster (153 vs. 52 WPM), and the Mandarin Chinese input rate was 2.87 times faster (123 vs. 43 WPM) than the keyboard for short message transcription under laboratory conditions for both methods. Furthermore, although speech made fewer errors during entry (5.30% vs. 11.22% corrected error rate), it left slightly more errors in the final transcribed text (1.30% vs. 0.79% uncorrected error rate). Our results show that comparatively, under ideal conditions for both methods, upper-bound speech recognition performance has greatly improved compared to prior systems, and might see greater uptake in the future, although further study is required to quantify performance in non-laboratory settings for both methods.
1 INTRODUCTION
The study addresses limited evidence comparing modern speech recognition with mobile keyboards and evaluates both methods under ideal conditions. Speech was substantially faster and made fewer errors during entry, but left slightly more errors afterward.
- Modern speech recognizers had not been adequately compared with improved smartphone keyboards, even under ideal conditions.
- 153 vs. 52 WPM in English and 123 vs. 43 WPM in Mandarin: speech was about 2.9 times faster than keyboard entry.
- 3.93% vs. 4.72% corrected error rate in English and 6.67% vs. 17.73% in Mandarin favored speech during entry.
- 0.55% vs. 0.35% uncorrected error rate in English and 2.06% vs. 1.22% in Mandarin left slightly more errors after speech entry.
- The work contributes an empirical comparison across English and Mandarin Chinese, a shared error-correction method, speech-specific measures, and interaction-design insights.
- Further research is needed to test speech and manual input under less-than-ideal conditions such as walking, ambient noise, and distraction.
2 RELATED WORK
Earlier mobile speech-entry studies reported low speed, high error, cascading correction failures, and user concerns. Recent advances in deep learning and computation motivated reevaluating speech against smartphone keyboards.
- Earlier studies reported mobile speech rates of 18 WPM seated and 13 WPM walking, while another found 13.6 WPM for transcription versus 32.5 WPM for keyboard-mouse input.
- About 33–44% recognition error rates were observed in mobile data entry on the move, partly associated with background noise.
- Speech-based correction commands can trigger cascading failures when the system misinterprets the correction itself.
- Users have reported dissatisfaction with speech input, including concerns about privacy, security, and confidentiality in social settings.
- Deep learning, increased data, and greater computational performance improved speech recognition enough to motivate renewed study of mobile speech entry.
3 EXPERIMENT
The experiment measured upper-bound performance for speech and touchscreen typing in English and Mandarin using a controlled laboratory setup. Participants used a custom iPhone test bed with speech recognition and QWERTY-based keyboards, including keyboard correction for speech.
- 3 EXPERIMENT: The controlled laboratory experiment evaluated upper-bound speed, accuracy, and low-level measures for speech recognition and touch-based keyboard entry in two languages.
- Participants: 48 university students participated: 24 native English speakers and 24 native Mandarin Chinese speakers, all familiar with the relevant iPhone keyboard.
- 3 EXPERIMENT: Participants were seated in a quiet room without walking or outside distractions, controlling factors beyond the study’s upper-bound scope.
- 3.2 Apparatus: The study used an Apple iPhone 6 Plus, Baidu Deep Speech 2, and Apple QWERTY keyboards for English and Mandarin Pinyin input.
- 3.2 Apparatus: Speech recognition displayed the utterance in a textbox, after which participants could switch to a keyboard for error correction or use speech again.
- 3.2 Apparatus: The study omitted advanced non-typing keyboards such as Swype and ShapeWriter.
3.3 Procedure
The procedure used balanced phrase sets and transcription tasks to isolate text-entry performance. Short, punctuation-free English and Mandarin phrases were assigned across keyboard and speech conditions to limit learning effects.
- 3.3 Procedure: Participants transcribed 60 phrases from one of two alternating phrase sets for each input method to avoid learning effects.
- 3.4 Text Transcription and the Phrase Set: Transcription was chosen over composition to separate input-method performance from thinking time, word choice, and writing ability.
- 3.4 Text Transcription and the Phrase Set: Transcription is less naturalistic than composition and may differentially affect speech because thinking can produce disfluencies that confuse recognizers.
- 3.4 Text Transcription and the Phrase Set: Shorter presented phrases reduce differences between transcription and composition and make speech disfluencies less concerning.
- Phrase Set: The English phrase set contained 120 phrases averaging 28.3 characters, with little capitalization and no punctuation.
- Phrase Set: English and Mandarin phrase sets corresponded one-to-one; Mandarin phrases ranged from 3 to 14 characters and averaged 7.7 characters.
3.5 Data Logging
The test-bed app logged user actions and timestamps during keyboard and speech trials, including speech-specific sessions and subsequent corrections.
- Data logging: The app logged pertinent user behaviors and timestamps for actions performed during each experiment trial.Logged actions included keystrokes and other input events.
- Data logging: The logging scheme classified input into insertion, deletion, auto-correction, word completion, and speech actions.Speech was available only in the speech-entry condition, whereas the other categories describe text-editing behavior.
- Data logging: Speech sessions recorded microphone activation, server responses, speaking turns, completion, and their client-side timestamps.A single trial could contain multiple speech sessions when users corrected text with voice or keyboard.
- Data logging: The English and Mandarin phrase sets were made available online so other researchers could build upon the study.The dataset contained 120 phrases in each language version.
3.6 Measures
The study measures text-entry speed and errors from logged transcriptions, input streams, and character-level edits, using language-specific word-length assumptions.
- Speed: Words per minute measures entry rate from trial duration and the transcribed string.Keyboard timing runs from the first to last character; speech timing runs from the first phoneme to the final user edit.
- Speed: English WPM treats words as five characters, while Mandarin Chinese uses an average word length of 1.5 characters.These values determine the language-specific conversion from characters to words.
- Input stream: The input stream is a sequence of strings that exposes insertions, deletions, whole-sentence replacement, and other corrections over time.The first string may be speech-system output, while the last string is the final transcription.
- Error rates: Error rates distinguish uncorrected errors remaining in the final text from corrected errors fixed during entry.Characters are classified as Correct, Incorrect-not-fixed, Incorrect-fixed, or Fixes.
- Error rates: The Correct class counts characters that are correct in the transcribed text using MAX(P,T) – MSD(P,T).MSD denotes minimum string distance, also called edit distance; P and T denote presented and transcribed strings.
Correct (C).
The error-analysis framework distinguishes incorrect final characters, deleted characters, and delete actions within the logged input stream.
- Incorrect final text: Incorrect final characters are counted as MSD(P,T), the minimum string distance between presented and transcribed strings.This definition concerns incorrect characters remaining in the final transcribed text.
- Deleted characters: Deleted characters are counted by summing the lengths of all deletion values.This measures characters removed during entry rather than only the number of deletion events.
- Example: Table 1 illustrates an input stream for transcribing a phrase with speech input and keyboard-based error corrections.The example phrase is “where did I leave my glasses.”
- Delete actions: Delete actions include single-character backspaces and whole-sentence deletion with the text-box X button.Pressing X counts as one fix action regardless of the deleted string’s length.
Fixes (F).
The study extends error and speed analysis with correction-related measures, initial speech-transcription timing, speech-specific efficiency measures, and subjective workload assessment.
- Correction measures: The study computes error rates and utilized bandwidth for both keyboard and speech input.These measures are derived from the classified input stream.
- Initial speech transcription: Initial speech transcription speed measures the speech system’s output before user corrections.The initial string is returned by the speech system before error correction, and its timing ends when the server returns the last character.
- Initial speech transcription: Initial speech-transcription error rates compare the initial speech string with the presented string before correction actions occur.For this analysis, the initial speech string is treated as the final string, so insertion and fix counts are zero.
- Speech-specific measures: Novel speech-specific measures evaluate efficiency and effectiveness, including speech session time and server process time.Speech session time spans phrase presentation through the final result; server process time spans speaking onset through that result.
- Subjective measures: Participants also rated workload with NASA TLX and discussed their subjective experiences of each input method.The workload instrument used a 7-point Likert scale across six categories.
3.7 Design & Analysis
The study used a 2×2 mixed factorial design to examine language and input-method effects, retaining trial-level observations for statistical analysis.
- Design: The experiment crossed input method within subjects with language between subjects in a 2×2 mixed factorial design.This design supported tests of main effects and interactions between language and input method.
- Design: Each participant completed separate speech and keyboard sessions, with session order counterbalanced across experiments.
- Data collection: Each participant contributed 50 speech trials and 50 keyboard trials, producing 4,800 trial-level data points across 48 log files.Keystrokes, timestamps, Pinyin, and resulting Mandarin characters were recorded where applicable.
- Analysis: Words per minute was analyzed with a parametric mixed-effects ANOVA using participant as a random effect.Error rates and other measures were analyzed with the nonparametric Aligned Rank Transform procedure.
4 RESULTS
Speech substantially outperformed the keyboard in entry speed across English and Mandarin, while producing fewer corrected errors but slightly more final uncorrected errors. Participants also reported lower workload and easier text production with speech, although speech performance included notable user and processing delays.
- Error Rates: Corrected error rates were lower for speech than keyboard input, especially in Mandarin: 6.67% versus 17.73% and 3.93% versus 4.72% in English.The difference was statistically significant for both languages and larger in Mandarin.
- Error Rates: Speech left more errors in final text than the keyboard, with uncorrected error rates of 1.30% versus 0.79%.The authors characterize this as a speed-accuracy tradeoff because speech was also faster.
- Speech-Specific Results: Only 39.2% of speech-input time was spent speaking, while user delays accounted for 31.5% and recognition processing for 22.6%.Delays included hesitation before speaking and uncertainty about pressing Done.
- Speech-Specific Results: Initial speech transcription produced 91.5% of speech-input task time, while users spent 8.5% correcting errors, mostly with the keyboard.Of correction time, 86.0% used the keyboard and 14.0% used speech; 96.7% of final characters came directly from initial speech input.
- Subjective Ratings: Participants rated speech as easier for producing text and less demanding than the keyboard across both languages.English speech received the lowest workload ratings across all six NASA TLX categories, while Mandarin keyboard received the highest difficulty ratings.
- Subjective Ratings: Participants described speech as easier to learn and use, less demanding, and more accurate than they had expected.
5 DISCUSSION
The study found that speech was nearly three times faster than keyboard input in both languages and made fewer errors during entry, but left slightly more uncorrected errors. These upper-bound results came from ideal laboratory conditions and motivate broader evaluations and interface improvements.
- Speed and accuracy: 2.93× faster in English and 2.87× faster in Mandarin, speech outperformed keyboard input in entry speed.English rates were 152.86 versus 52.24 WPM; Mandarin rates were 123.00 versus 42.83 WPM.
- Speed and accuracy: Speech made fewer errors during entry but left slightly more uncorrected errors afterward.The authors attribute this tradeoff partly to differences in when users could notice and correct errors.
- Speech processing: 4.37% English and 9.94% Mandarin uncorrected error rates came from the initial Deep Speech 2 transcription.Users therefore needed only a few corrections after the initial speech transcription.
- User experience: Participants generally preferred speech, describing it as more natural, smooth, and immediately responsive for novice users.The study used only 10 practice phrases before testing.
- Potential improvements: 22.6% of experiment time was speech processing, while user delay accounted for 31.5%.Reducing end-of-speech delay or automatically detecting when users finish speaking could improve speed.
- Limitations and future work: The comparison estimates upper-bound performance because speech used a quiet, fast-network environment and keyboard users were seated and familiar with keyboards.The authors call for studies under conditions such as walking, ambient noise, and distraction.
6 CONCLUSION
This study compared state-of-the-art speech and touchscreen keyboard text entry in English and Mandarin Chinese under controlled laboratory conditions. Speech was nearly three times faster and made fewer errors during entry, but left slightly more errors afterward, establishing a baseline for future studies.
- Conclusion: Speech was nearly three times faster than the keyboard and made fewer errors during entry, but left slightly more errors afterward.The comparison covered English and Mandarin Chinese under ideal conditions for both methods.
- Conclusion: The study provides a baseline for comparing speech-based mobile text entry in future in-the-wild evaluations.It also introduces speech-specific measures for evaluating mobile speech input.