PaperlayerView exact revision on arXiv

Source-linked AI summary

LUMO (Lightweight Unified Multilingual Orchestrator): A Privacy Preserving Offline Voice Assistant

Md. Mehedi Hasan Naeem, Mst. Kamrunnahar Ruma, Nafiza Anjum, Shakila Sultana, Md. Sujan Ali

arXiv:2609.30692v1cs.LG

TL;DR

Cloud-based voice assistants raise concerns about latency, internet dependence, and privacy, while existing edge systems have limited conversational capability and system-level evaluation. LUMO combines offline speech recognition, a quantized local LLM, and speech synthesis on Raspberry Pi; it reports 3.2 s latency and 9 W power consumption, with local processing intended to preserve privacy.

  • Problem

    Existing voice assistants often depend on cloud services, while many edge assistants offer limited open-ended conversation and lack complete system-level evaluation.

    Source: §I · p. 1, §II · p. 2

  • Method

    LUMO integrates on-device speech recognition, a quantized lightweight local language model, and text-to-speech in an offline Raspberry Pi pipeline.

    Source: §II · p. 2, A. System Overview · p. 2

  • Results

    3.2 s latency and 9 W power consumption; LUMO supports local generative LLM-based reasoning unlike the compared systems.

    Source: H. Comparison with Rhasspy and Mycroft · p. 6

  • Takeaways & Limitations

    Local processing supports private operation while reducing dependence on network connectivity.

    Source: A. System Overview · p. 2

    Reported limitations include performance in highly noisy environments, limited advanced reasoning, voice-only interaction, and no IoT context or long-term memory.

    Source: H. Comparison with Rhasspy and Mycroft · p. 6

Abstract

from arXiv · show

Reliable voice interaction is essential in environments with limited internet connectivity and strong privacy. However, most existing voice assistants depend on cloud-based services, which leads to latency issues, dependency on internet access, and privacy vulnerabilities. This research presents LUMO (Lightweight Unified Multilingual Orchestrator), a privacy preserving offline voice assistant designed for edge computing environments. This system integrates local Automatic Speech Recognition (ASR), locally deployed quantized Large Language Model (LLM), and Text-to-Speech (TTS) synthesis into a fully offline pipeline running on a Raspberry Pi 5 with 8 GB RAM. To enable efficient operation on resource constrained hardware, the language model is compressed using 4-bit GGUF quantization, which reduces memory usage while preserving practical conversational capability. Existing edge based voice assistants Mycroft provides partial offline functionality without a generative LLM, with an approximate latency of ~5 s and power consumption of ~12 W, while Rhasspy supports full offline operation but lacks generative capabilities, with ~3 s latency and ~11 W power usage. In contrast, LUMO achieves a Word Error Rate (WER) of 6.8% for short English utterances in low noise conditions, an end-to-end response latency of 2.0-4.0 s, and a lower peak power consumption of approximately 9.0 W. The system also achieves effective offline recognition for Bangla speech, supporting multilingual accessibility in low resource settings. By operating entirely offline, LUMO provides strong data privacy, reduced need for cloud connectivity, and suitability for privacy sensitive edge execution such as rural healthcare, education, and disaster response scenarios.

I. INTRODUCTION

Cloud-dependent assistants raise latency, connectivity, and privacy concerns, while edge systems still face constraints on open-ended conversation and complete system evaluation. LUMO integrates offline speech recognition, a quantized local language model, and speech synthesis for Raspberry Pi.

I. INTRODUCTION

  • Cloud-based assistants depend on networks and can introduce latency and privacy concerns; edge deployment must balance inference speed, memory use, and response quality.The paper also identifies limited open-ended conversation and insufficient evaluation of latency, thermal behavior, and power consumption as gaps in existing systems.Source: §I · p. 1

II. BACKGROUND STUDY AND RELATED WORK

  • Prior work covers edge AI, lightweight language models, quantization, and offline multilingual ASR, but many assistants remain rule-based and system-level evaluation is limited.The related-work discussion surveys embedded LLM deployment and speech-recognition resources while identifying incomplete offline assistant integration as a gap.Source: §II · pp. 1–2

III. PROPOSED SYSTEM DESIGN AND METHODOLOGY

  • LUMO combines on-device speech recognition, a quantized lightweight language model, and text-to-speech in a unified multilingual architecture for resource-constrained Raspberry Pi devices.The system-design section describes this as an offline edge framework integrating speech processing, language reasoning, and synthesis.Source: §II · p. 2, §III · p. 2

A. System Overview

LUMO runs a local VAD-to-speech pipeline on Raspberry Pi 5, with modular processing and an evaluation dataset of short English and Bangla commands.

A. System Overview

  • LUMO uses a Raspberry Pi 5 pipeline of voice activity detection, speech-to-text, a local language model, and text-to-speech, with all processing performed locally.Local processing is described as preserving privacy and reducing network dependence.Source: A. System Overview · p. 2

B. Multilingual Speech Dataset Design

  • The evaluation dataset comprises short command-style English and Bangla utterances recorded by ten bilingual speakers and manually transcribed.The recordings used the same USB condenser microphone and deployment environment; pretrained models were evaluated without task-specific fine-tuning.Source: B. Multilingual Speech Dataset Design · p. 2

C. Hardware Configuration

  • LUMO runs on a Raspberry Pi 5 with 8 GB RAM, using a USB condenser microphone and speaker for audio input and output.A NeoPixel LED ring indicates listening, processing, and speaking states.Source: C. Hardware Configuration · p. 2

D. Software Pipeline

  • Audio is captured in approximately 0.25-second frames, analyzed by VAD, then transcribed, processed by the LLM, and converted to speech by TTS.The modular design allows independent component optimization and supports edge deployment.Source: D. Software Pipeline · p. 2

E. Voice Activity Detection and Speech Recognition

LUMO uses offline speech recognition, local quantized LLM inference, and offline speech synthesis, with queued multithreaded modules supporting low-latency interaction.

E. Voice Activity Detection and Speech Recognition

F. Local LLM Integration

  • Recognized text goes to a locally deployed GGUF-quantized LLM, whose lower-precision weights reduce memory requirements while supporting responses without cloud transmission.The passage describes local inference as providing context-aware responses.Source: F. Local LLM Integration · p. 2

G. Text-to-Speech Module

  • Piper converts generated responses into speech with low computational requirements, and offline TTS engines support speech generation without internet connectivity.The operational states are listening, processing, and speaking.Source: G. Text-to-Speech Module · p. 3

H. Concurrency and Control Logic

  • Separate threads handle audio capture, recognition, LLM inference, and playback, passing audio and text through queues to reduce blocking and support low-latency interaction.The multithreaded architecture is identified as the mechanism for coordinating these modules.Source: H. Concurrency and Control Logic · p. 3

I. Deployment Optimization

LUMO targets efficient offline operation on constrained hardware through quantization, short audio frames, optimized storage, and active cooling; evaluation spans recognition and system behavior.

I. Deployment Optimization

  • Quantization reduces memory and computational requirements, while short audio frames reduce processing latency.USB SSD storage improves model loading speed, and active cooling maintains stable temperature during prolonged inference.Source: §I · p. 3

IV. EXPERIMENTAL SETUP

  • LUMO was evaluated offline on resource-constrained hardware using recognition accuracy, response latency, memory, power, and thermal behavior as primary metrics.The experiments were designed to reflect realistic operation without internet connectivity.Source: §IV · p. 3

A. Hardware and Software Environment

  • The deployment uses a Raspberry Pi 5 with an ARM Cortex-A76 processor, 8 GB RAM, and 64-bit Raspberry Pi OS, integrating VOSK, WebRTC, GPT4All, and Piper.Audio input and output use a USB condenser microphone and standard speaker.Source: A. Hardware and Software Environment · p. 3

B. Speech Dataset

  • A custom dataset contains 1,000 command-style utterances: 500 English and 500 Bangla recordings from ten bilingual speakers, stored as 16-kHz mono WAV files.Ground-truth transcriptions were manually verified, and the dataset and implementation are publicly available.Source: B. Speech Dataset · p. 3

C. Noise-Level Test Conditions

The evaluation tests speech recognition under three ambient-noise conditions and assesses recognition, latency, resource use, and thermal stability. It defines WER and end-to-end latency as evaluation measures.

C. Noise-Level Test Conditions

  • Commands were tested in quiet (< 30 dB), moderate (45–60 dB), and high (70–75 dB) ambient noise, representing indoor, office, and crowded environments.The same speech commands were evaluated at each noise level.Source: C. Noise-Level Test Conditions · p. 4

D. Evaluation Metrics

  • WER counts substitution, deletion, and insertion errors relative to the number of reference words; lower WER indicates better recognition.Source: D. Evaluation Metrics · p. 4
  • End-to-end latency is measured from the end of user speech to the start of synthesized audio, alongside memory, power, and temperature monitoring.The monitored resource measures assess efficiency and thermal stability.Source: D. Evaluation Metrics · p. 4

E. Privacy Preservation and Cloud Independence

LUMO evaluates cloud independence using network traffic and external data exposure, reporting both as zero during offline operation. The section also compares assistant frameworks conceptually and reports recognition performance across noise levels.

E. Privacy Preservation and Cloud Independence

F. Baseline Systems for Comparison

  • LUMO combines offline speech processing with a quantized local LLM, while Rhasspy and Mycroft primarily use predefined intents, command grammars, or rule-based processing.Source: F. Baseline Systems for Comparison · p. 4

V. RESULTS AND DISCUSSION

  • The results section evaluates speech recognition, multilingual capability, LLM performance, latency, power, thermal stability, and offline operation on Raspberry Pi 5.Source: §V · p. 4

A. Offline Speech Recognition Performance

B. Multilingual Performance Analysis

LUMO recognizes both English and Bangla speech offline, with Bangla showing a slightly higher error rate. TinyLLaMA was selected for its balance of inference speed and memory requirements.

B. Multilingual Performance Analysis

  • LUMO supports English and Bangla speech recognition entirely offline, though Bangla has a slightly higher error rate attributed to limited training resources for low-resource languages [21].Source: B. Multilingual Performance Analysis · p. 4

C. Lightweight LLM Benchmark and Model Selection

D. Quantization Impact on Memory and Speed

The section reports the model-size reduction from 4-bit GGUF quantization and breaks down end-to-end response latency, identifying LLM inference as the largest latency component.

D. Quantization Impact on Memory and Speed

E. End-to-End Latency Analysis

F. Power Consumption Analysis

LUMO’s measured power draw was 4.2 W at idle, 6.8 W during speech recognition, and 9.0 W during LLM inference. Compared with Rhasspy and Mycroft, its reported consumption was lower, while CPU temperature remained below 70°C during inference.

F. Power Consumption Analysis

G. Thermal Stability Analysis

  • CPU temperature rose gradually during LLM inference but remained below 70°C, indicating stable operation without sudden thermal degradation.Fig. 8 presents the CPU temperature change during continual LLM inference.Source: G. Thermal Stability Analysis · p. 5, Figure 8 · p. 5

H. Comparison with Rhasspy and Mycroft

Against Rhasspy and Mycroft, LUMO combines on-device execution and local generative reasoning with lower reported power use and faster latency. The conclusion presents it as feasible for edge devices while noting limits in noisy conditions and compact-model reasoning.

H. Comparison with Rhasspy and Mycroft

  • LUMO operates entirely on-device with zero outbound traffic during offline execution, preventing conversational data from being transmitted to external services.Table VIII covers data privacy and cloud connectivity.Source: H. Comparison with Rhasspy and Mycroft · p. 6, Table VIII · p. 6
  • 3.2 s latency improved by 36.0% over Mycroft and 8.6% over Rhasspy; LUMO also used 9 W versus 11 W and 12 W, respectively.Unlike the compared systems, LUMO supports local generative LLM-based reasoning for more flexible conversational interaction.Source: H. Comparison with Rhasspy and Mycroft · p. 6, Table IX · p. 6
  • Current limitations include performance in highly noisy environments, limited advanced reasoning, voice-only interaction, and no IoT context or long-term memory.The conclusion also notes comparatively higher Bangla WER and limited evaluation on longer multi-turn conversations.Source: H. Comparison with Rhasspy and Mycroft · p. 6, §VI · p. 6

VI. CONCLUSION

  • The paper concludes that privacy-preserving conversational assistants can be deployed on low-power edge devices through model optimization and efficient pipeline integration.Future work includes evaluating conversational scalability under edge-hardware constraints.Source: §VI · p. 6
Loading 2609.30692v1…