Source-linked AI summary
SottoVoce: An Ultrasound Imaging-Based Silent Speech Interaction Using Deep Neural Networks
Naoki Kimura, Michinari Kono, Jun Rekimoto
TL;DR
Speaking aloud in public can annoy others or disclose private information, and environmental noise may reduce speech-recognition accuracy. SottoVoce uses an ultrasonic sensor beneath the jaw and two neural networks to convert sequences of ultrasound images into audio. The method could operate existing voice-controllable devices such as smart speakers; WER was 20.61%, 41.03%, and 33.56% for GT, Network 1, and Network 2.
Problem
Speaking aloud in public can annoy others or disclose private information, and environmental noise may reduce speech-recognition accuracy.
Source: §1 · p. 2Method
Results
Takeaways & Limitations
The authors suggest incremental audio generation could give users tighter feedback for learning oral movements that improve generated speech.
Source: §6.1 · p. 7Continuous ultrasonic emission's effect on human organs is unknown, and current output is generated at whole-command granularity.
Source: §6.2 · p. 7, §6.1 · p. 7
Abstract
from arXiv · showhide
The availability of digital devices operated by voice is expanding rapidly. However, the applications of voice interfaces are still restricted. For example, speaking in public places becomes an annoyance to the surrounding people, and secret information should not be uttered. Environmental noise may reduce the accuracy of speech recognition. To address these limitations, a system to detect a user's unvoiced utterance is proposed. From internal information observed by an ultrasonic imaging sensor attached to the underside of the jaw, our proposed system recognizes the utterance contents without the user's uttering voice. Our proposed deep neural network model is used to obtain acoustic features from a sequence of ultrasound images. We confirmed that audio signals generated by our system can control the existing smart speakers. We also observed that a user can adjust their oral movement to learn and improve the accuracy of their voice recognition.
1 INTRODUCTION
Voice interfaces are useful across devices and tasks, but speaking aloud can expose private information, disturb others, or fail in noise. SottoVoce uses ultrasound beneath the jaw and deep neural networks to turn silent mouth movements into audio, with proof-of-concept control of an unchanged smart speaker.
1 INTRODUCTION
- Public speech can annoy bystanders or disclose private information, while environmental noise may reduce recognition accuracy.These constraints are especially acute for wearables and mobile computers.Source: §1 · p. 2
- Earlier silent-speech approaches use face cameras, noticeable whispering, or EMG, which can limit wearability, privacy, or free-utterance recognition.The paper notes that EMG systems may limit detectable commands and require users to learn new gesture skills.Source: §1 · p. 2
- Ultrasound beneath the jaw could sense internal oral movement, including tongue motion, using a miniaturizable sensor suitable for inconspicuous wearable designs.Internal tongue movement cannot be observed externally and may support more accurate sound reproduction.Source: §1 · p. 2
- SottoVoce addresses prior ultrasound systems' reliance on face cameras and validates convolutional networks with actual silent speech and an unchanged smart speaker.Earlier deep-network ultrasound work was validated while users were voicing, rather than silently mouthing, utterances.Source: §1 · p. 2
- The system converts a user's mouthed utterance into sound using two deep neural networks and can control an existing voice-controlled device without modification.The proof of concept used Amazon Echo.Source: §1 · p. 2
2 RELATED WORK
Related work spans silent-speech sensing, non-auditory interaction devices, and smart-agent interfaces. The paper positions SottoVoce within those directions while emphasizing actual device interaction and user adaptation.
2.1 Silent Speech
- Silent-speech work spans lip or face imaging, wearable physiological sensing, and ultrasound; prior ultrasound systems often combine optical images, while deep-network work remained proof-of-concept without smart-device control.The authors aim to connect silent speech to voice-controlled devices and to improve performance through human–network interaction.Source: §2.1 · pp. 2–3
2.2 Non-Auditory Inputs and Interaction
- Non-auditory interaction includes electric-field, ear-canal pressure, radar, EMG, and brain–muscle sensing; EchoFlex uses ultrasound to recognize forearm-muscle movement.EchoFlex inspired consideration of ultrasound for sensing inside the mouth.Source: §2.2 · p. 3
2.3 Interacting with Smart Devices
- Smart-device interaction includes voice agents such as Amazon Echo, while prior Glove Talk II translated hand gestures into speech controls but required about 100 hours of training.The paper frames its approach as human–AI interaction in which users learn to adapt to the system.Source: §2.3 · p. 3
3 SYSTEM ARCHITECTURE OF SOTTOVOCE
SottoVoce maps under-jaw ultrasound image sequences to speech through two speaker-dependent neural networks. The system is trained on paired ultrasound videos and spoken commands, then converts predicted Mel-scale vectors into audio.
3 SYSTEM ARCHITECTURE OF SOTTOVOCE
- Network 1 maps ultrasound-image sequences to Mel-scale sound vectors, and Network 2 refines those vectors before they are converted into audio.The first network predicts 64-dimensional vectors; the second outputs a sequence of vectors with the same length as its input.Source: §3 · p. 3
3.1 Ultrasonic Imaging Device
- A 3.5-MHz convex probe is attached under the jaw, and a 300-ms delay between ultrasound images and sound capture is compensated in the training data.The device output is digitized into an MPEG-4 movie file.Source: §3.1 · p. 3
3.2 Network 1
- Network 1 uses 13 monochrome 128 × 128 ultrasound images spanning 400 ms to predict a Mel-scale spectrum every 20 ms.Its inputs and outputs are normalized to 0–1, with mean-squared error loss and the Adam optimizer.Source: §3.2 · pp. 3–4
3.3 Network 2
- Network 2 refines Mel-scale sequences using a 1-D convolutional filter bank and a U-Net, trained on Network 1 predictions paired with corresponding audio-derived vectors.Input and output durations are fixed at 3.68 seconds.Source: §3.3 · pp. 4–5
3.4 Sound generation
- The predicted Mel-scale vectors are converted to audio with the Griffin Lim algorithm, and the generated sound can be played through a speaker to control nearby voice-operated devices.Conversion can use output from either Network 1 or Network 2.Source: §3.4 · p. 5
3.5 Training
- Two collaborators each supplied approximately 500 spoken commands with corresponding ultrasound video; both speaker-dependent networks were trained separately for each speaker.Network 1 was trained first and used to create Network 2's training inputs; Gaussian noise was applied to Mel-scale vectors to augment Network 2's test sets.Source: §3.5 · p. 5
3.6 Implementation Details
- For a 3.68-second input, total processing—including video handling, neural-network processing, and Mel-to-audio conversion—took 2.61 seconds.Neural-network processing alone required 2.36 seconds.Source: §3.6 · p. 6
4 RESULTS
Network 1 plus Network 2 improved smart-speaker command recognition over Network 1 alone, and the generated audio controlled unchanged smart speakers. The results also show that Network 2 had lower word error rate than Network 1, while longer commands were easier to recognize than shorter ones.
4 RESULTS
- Both networks generated natural intonation, which the authors suggest may indicate that the networks learn speech context.Source: §4 · p. 6
- Network 1 plus Network 2 improves smart-speaker recognition over Network 1, and generated audio controlled unchanged Amazon Echo and Echo Show devices.The comparison is reported in Table 1; the text does not provide its success-ratio values.Source: §4 · p. 6, Table 1 · p. 6
- 20.61% WER for GT, 41.03% for Network 1, and 33.56% for Network 2 were measured using Google’s cloud speech-to-text engine.The values are means across 40 speech commands from two users.Source: §4 · p. 6
- Longer commands such as “what’s the weather like?” had higher recognition rates across conditions than shorter commands such as “play jazz.”The authors suggest longer commands may provide Network 2 more context.Source: §4 · p. 6
5 END-TO-END EVALUATION AND OBSERVATIONS
End-to-end tests translated mouthed commands into generated audio, but voiceless-input output initially sounded worse than voiced-input output. Listening to generated audio let users adjust their mouth movements, after which sound quality improved.
5 END-TO-END EVALUATION AND OBSERVATIONS
- The end-to-end test recorded oral-cavity movement while users mouthed commands without intentionally emitting sound, then translated the image sequences into voice.Participants were asked not to vibrate their vocal cords, but they were not asked to hold their breath; mean peak sound level across 20 measurements was 37.14 dB(A), lower than soft whispering.Source: §5 · p. 6
- Voiceless-input audio was initially lower in quality than voiced-input audio, but users improved it after listening and adjusting their mouth movements.Figure 10 compares Network 2 output from voiced and voiceless ultrasound images against the original encoded-and-decoded voice.Source: §5 · p. 6, Figure 10 · p. 7
- Figure 9 presents training results as waves and Mel-scale spectrograms for Network 1 and Network 2, alongside the original voice encoded by a Mel-scale spectrogram and decoded to audio as training ground truth.Source: Figure 9 · p. 7
6 DISCUSSIONS
The discussion identifies limits in output timing and continuous ultrasound exposure, while proposing future uses and multimodal extensions. It also frames user learning through generated-audio feedback as human–AI integration.
6.1 Incremental Voice Generation
- Network 2's fixed-length voice representation limits current output to speech-command chunks of approximately 3.6 seconds.The authors suggest incremental generation to provide a tighter feedback loop for learning oral movements.Source: §6.1 · p. 7
6.2 Continuous Ultrasonic-Wave Emission into the Body
- The effects of continuous ultrasound emission on human organs are unknown; an accelerometer and microphone could trigger emission to start and stop.This is offered as a possible way to avoid continuous emission.Source: §6.2 · p. 7
6.3 Cure for Vocal Cord Disabilities
- The authors expect people with damaged vocal cords could use the system to learn mouth and tongue control for generating sound.Source: §6.3 · p. 8
6.4 Combining with Other Modalities
- Combining ultrasound with EMG, accelerometers, and NAM microphones may improve speech-recognition quality; investigating these combinations is future work.Source: §6.4 · p. 8
6.5 Human–AI integration
- The authors call user learning through a feedback loop between organic and artificial neural networks “human–AI integration.”They relate this loop to speech learning and cite Glove Talk II, whose pianist user needed more than 100 hours to produce audible voice with hand gestures.Source: §6.5 · p. 8
7 CONCLUSION
SottoVoce uses two neural networks in sequence to turn a user's mouthed, unvoiced utterance into sound. The authors envision pairing an under-jaw ultrasound probe with an open-air or bone-conduction earphone for wearable voice-assistant use.
7 CONCLUSION
- Two neural networks convert a mouthed utterance without voice, captured through ultrasound imaging, into sound that can operate existing voice-controlled devices.Source: §7 · p. 8
- The envisioned wearable pairs an attachable under-jaw ultrasound probe with a bone-conduction or open-air earphone, allowing users to invoke an assistant silently and receive responses.Source: §7 · p. 8