Source-linked AI summary
Your Home Remembers: A Voice-Enabled Household Memory Assistant for Conversational Retrieval
Aleix Sant, Jordi Luque, Fernando López, Pablo Gómez, Carlos Escolano
TL;DR
Household details exchanged in conversation are often not written down and can be difficult to recall. This paper presents a voice-enabled assistant that indexes recorded conversations as timestamped, turn-aware passages and retrieves them for spoken queries. The resulting assistant makes stored and recently captured conversations accessible through spoken interaction.
Problem
Household information exchanged in conversation is often not written down, making specific details difficult to recall.
Source: §1 · p. 1Method
The assistant transcribes and diarises recorded speech into timestamped, turn-aware passages, indexes them, and retrieves relevant passages to ground generated answers.
Source: §1 · p. 1, §3.3 · p. 2, §3.4 · p. 3Results
The resulting assistant makes stored and recently captured conversations accessible through spoken queries.
Source: §5 · p. 3Takeaways & Limitations
The assistant is intended to help users retrieve what was actually said, not replace a clinician or provide new medical advice.
Source: §1 · p. 1Grounding can improve contextual relevance but does not guarantee factual correctness because retrieval, generation, or the recorded source may contain errors.
Source: §3.4 · p. 3
Abstract
from arXiv · showhide
Everyday household information is often exchanged in conversations but never written down, making details easy to forget or misremember. We present a voice-enabled smart-home assistant that acts as a household memory by recording conversations and retrieving information from stored and newly captured speech. The assistant runs on an Android tablet with a modular architecture in which speech-recognition, diarisation, embedding and language models can be configured or replaced through an orchestration layer. On-device wake-word detection and VAD-based query capture support hands-free interaction. Recorded audio is processed in configurable chunks, transcribed and diarised, converted into turn-aware passages with conversation-level timestamps, and indexed in a vector store. For memory queries, a retrieval-augmented generation pipeline selects key passages and uses them to generate an answer. The interface supports recording, conversation management and inspection of timestamped sources.
1. Introduction
Household details exchanged in conversation can be difficult to recover, especially across people, topics, and decisions. The assistant turns consented conversations into searchable memory and retrieves answers from selected recorded speech.
1. Introduction
- The assistant grounds responses in conversations recorded and selected by the household, rather than relying primarily on parametric knowledge.Retrieval-augmented generation supports queries over both stored conversations and speech processed shortly before the query.Source: §1 · p. 1
- A medical-recall scenario calls for preserving speaker attribution and temporal provenance while distinguishing conversational evidence from generated wording.The assistant is intended to retrieve what was said, not replace a clinician, interpret symptoms, or provide new medical advice.Source: §1 · p. 1
- The system combines turn-aware passages with conversation-level timestamps and a tablet interface for inspecting the passages used as answer evidence.Its contributions also include modular orchestration that allows compatible speech and language models to be selected or integrated without changing interaction logic.Source: §1 · p. 1
2. Assistant interaction
The interface supports creating and managing conversational memories, then querying a selected memory by voice. Users can inspect retrieved source snippets and adjust retrieval and response settings.
2. Assistant interaction
- Users can create named recordings, choose audio-chunk duration, select or delete stored conversations, and clear short-term dialogue history without deleting indexed memories.The interface displays capture and processing status and provides an explicit stop control to end recording and ingest the final chunk.Source: §2 · p. 1
- A wake word activates hands-free queries; the assistant captures speech, retrieves relevant passages, displays an answer, and lists timestamped source snippets for inspection.The prototype supports Spanish and English.Source: §2 · p. 1
- Users can configure retrieval, response, language, embedding, passage-splitting, and model settings to balance latency, precision, answer length, and computational requirements.The remote orchestrator routes requests to configured model and memory services through a common backend interface.Source: §2 · p. 1, Figure 1 · p. 2
3. Technical architecture
The system separates on-device interaction from remote model and memory services. Its pipeline captures queries, processes conversations into timestamped passages, and retrieves passages to ground generated answers.
3. Technical architecture
- A tablet query path captures speech before remote transcription, retrieval, and answer generation, while a separate memory path ingests configurable audio chunks into a conversation-specific vector index.This separation keeps latency-sensitive interaction on-device while speech, retrieval, and language components can be configured as independent services.Source: §3 · p. 2
3.1. Orchestration and modular architecture
- An orchestration layer validates task requests and routes them through common adapters, allowing registered models to be selected or replaced without changing Android interaction logic.The current configuration uses separate models for short-query transcription, conversation transcription and diarisation, answer generation, and embeddings.Source: §3.1 · p. 2
3.2. On-device audio interaction
- A local GRU-based wake-word detector activates tablet audio capture; VAD then marks query endpoints before the utterance is sent to the selected ASR service.Query capture uses a speech-probability threshold of 0.5 and a two-second silence interval; long-form recording is handled separately in configurable chunks.Source: §3.2 · p. 2
3.3. Turn-aware conversational memory
- Timestamp conversion, bounded same-speaker merging, and speaker reconciliation produce timestamped turns and passages, though turns spanning successive audio chunks are not rejoined.The merge rule allows gaps from −0.2 to 1.0 seconds; passages retain speaker-labelled text, start and end times, and turn counts.Source: §3.3 · pp. 2–3
3.4. Vector store, retrieval and response
- Conversation-specific LanceDB indexes store configurable passage embeddings; query-time nearest passages are supplied as generation context, but grounding does not guarantee factual correctness.Passage metadata, including timestamps and turn counts, is returned separately for display and excluded from the language-model prompt.Source: §3.4 · p. 3
4. Ethical considerations
The paper identifies additional safeguards needed for real-world deployment, including participant consent, privacy controls, access protections, and answer guardrails.
4. Ethical considerations
- Real-world deployment would require informed consent from all participants, persistent recording notifications, private modes, clear retention policies, and secure data storage.Source: §4 · p. 3
- Future versions should provide authentication, per-user permissions, and speaker verification to reduce the risk of cross-resident access or mixed records.Source: §4 · p. 3
- The assistant should communicate uncertainty and decline to answer when reliable evidence is unavailable, and should be presented as a recall aid rather than an authority.Suggested guardrails include retrieval-confidence thresholds and consistency checks.Source: §4 · p. 3
5. Conclusion
The assistant makes stored and recently captured conversations accessible through spoken queries. Its modular architecture preserves speaker and temporal context and supports inspection of timestamped evidence.
5. Conclusion
- Stored and recently captured conversations are accessible through spoken queries.The assistant is intended to make household conversational memory retrievable by voice.Source: §5 · p. 3
- The modular architecture separates the Android interaction layer, model services, and conversational-memory pipeline.Compatible speech, embedding, and language models can be selected or integrated without changing the assistant interaction logic.Source: §5 · p. 3
- Turn-aware ingestion preserves speaker and temporal context, while the interface supports conversation management and inspection of timestamped passages used to support answers.Source: §5 · p. 3