Scalable Context Orchestration for Serving LLMs Over Voice

• Voice context spans semantic content, paralinguistic cues, and environmental conditions, but current context management mainly handles semantics and leaves the other states implicit.Source: §1 · p. 1
• Runtime decisions must jointly account for meaning, delivery, and conditions, since explicit user instructions can conflict with cues such as speaking rate or apparent silence.Source: §1 · p. 2
• Appending growing voice history makes context selection ineffective and processing costs cumulative, with repeated history processing increasing total session cost quadratically.Source: §1 · p. 2
• llmovoice constructs bounded context from current input, relevant history, and explicit states, then uses the serving LLM to generate runtime directives and a response.Source: §1 · p. 2
• 52.4% lower speaking-rate alignment error, false-interruption rate fell from 46.0% to 0.9% under packet loss, and network-induced cost fell 79.2%.Source: §1 · pp. 2–3
• For long conversations, per-turn cost fell up to 24.9× while retaining up to 98.7% of baseline answer quality.Source: §1 · p. 3

2.1 LLM-over-Voice Serving Pipeline

• Real-time serving streams audio chunks, detects when the user stops speaking, invokes the model with the utterance and history, and streams its response back.Source: §2.1 · p. 3

2.2 Limitations of Existing Systems

• Existing systems leave paralinguistic and environmental states implicit, while separate state-specific controllers can make conflicting runtime changes.Source: §2.2 · p. 3
• A packet-loss burst can push decoded audio below the VAD energy threshold long enough to trigger a false endpoint and prematurely invoke the model.Source: §2.2 · p. 3, Figure 2 · p. 3
• Per-turn cost grows approximately linearly with conversation history under different cache settings, leaving total session cost quadratic and context capacity unchanged.Source: §2.2 · pp. 3–4, Figure 3 · p. 3

3.1 Architecture Overview

• llmovoice combines projected history with the current input, then uses the serving LLM to reason over voice states and issue validated runtime directives.Completed interactions are committed to memory for use in subsequent turns.Source: §3.1 · p. 4

3.2 Voice Context Modeling

• Voice context separates semantic content, speech delivery, and capture or transmission conditions, with speaking rate and packet-loss telemetry as representative signals.The system maintains recent speaking-rate observations and aligns network observations with the received-audio timeline to help distinguish impairment from user behavior.Source: §3.2 · p. 4

3.3 Context Orchestration

• The orchestrator jointly reasons over voice states and runtime configuration, validating function calls that can modify response content, delivery style, or turn-taking.It may issue no directive, one, or multiple directives; orchestration adds cost and latency when invoked, but may avoid unnecessary downstream actions.Source: §3.3 · p. 5

3.4 Structured Voice Memory

• A vpage stores a bounded interaction and its available representations, while a vthread organizes related pages into a conversational flow that can span nonadjacent turns.Pages can belong to multiple threads; when no thread matches, page-level retrieval can recover details and seed a new thread.Source: §3.4 · pp. 5–6

3.5 Multi-Fidelity Context Projection

• The projector selects which history to include and whether to represent it as audio, transcript, or summary, since these forms retain different levels of information at different costs.The layered-degradation policy starts at each candidate’s highest available fidelity and reduces fidelity in increasing order of relevance until the context fits its budget.Source: §3.5 · pp. 6–7
• Under a context budget, the projector retrieves relevant threads and pages, then degrades lower-relevance representations from audio to transcript to summary or omission.The policy has O(N log N) time complexity, and scheduling overhead is around 1 ms.Source: §3.5 · pp. 6–7

4 Llmovoice Implementation

• The 1,246-line prototype connects browser WebRTC audio to a Python backend that relays audio to the OpenAI Realtime API and executes orchestration decisions.It uses PostgreSQL and pgvector for persistent conversation metadata and retrieval indexes, and the authors note it can be ported to other streaming model APIs.Source: §4 · p. 7

4.1 State-to-Control Path

• Speaking-rate state uses a linearly weighted average of recent valid utterance rates, smoothing turn-level variation while following sustained changes.Utterances shorter than 0.5 seconds and English observations outside 100–300 words per minute are discarded; the estimator retains the ten most recent valid observations.Source: §4.1 · p. 7
• The receiver samples WebRTC statistics every 50 ms and tracks concealed-audio duration, silence concealment, and time since the latest concealment over a rolling window.The window length is max{1000 ms, D_VAD}, where D_VAD is the current silence threshold.Source: §4.1 · p. 7
• The serving LLM jointly selects content, delivery, deferral, and turn-detection controls, while the runtime validates typed function calls before applying them.A deferred response is reconsidered after a 1.5-second soft deadline and forced after a 5-second hard deadline; subsequent speech can confirm a premature boundary and trigger an automatic VAD update.Source: §4.1 · pp. 7–8

4.2 Memory-to-Context Path

• The memory-to-context path retrieves using summaries and cosine similarity, while estimating audio, transcript, and summary input costs to schedule selected representations.Shared pages are deduplicated when threads expand into pages; compact thread summaries are costed directly from their word counts.Source: §4.2 · p. 8

5 Evaluation

• Across five research questions, llmovoice improves voice control and robustness, bounds long-session costs, and preserves answer quality, with modest context-organization overhead.The evaluation also reports real-world cases involving topic resumption, network-aware turn handling, and recovery after disconnection.Source: §5 · p. 8

5.1 Experimental Setup

• Experiments use RT2, MSP-PODCAST, and ICASSP PLC traces, with 120 RT2 and 138 MSP-PODCAST long-context queries and shared cost budgets for bounded-context comparisons.The setup compares multiple retrieval and compression baselines and measures alignment, false interruptions, network-induced cost, retrieval, answer quality, and TTFA.Source: §5.1 · p. 9

5.2 Runtime Control with Voice Context

• 52.4% lower speaking-rate alignment error and 98.0% lower false-interruption rate accompany a 79.2% reduction in network-induced cost under packet loss.Fixed-500 yields a 46.0% false-interruption rate versus 0.9% for llmovoice; Static-Max eliminates false interruptions but has 4.02 s median endpoint latency versus 1.83 s for llmovoice.Source: Table 1 · p. 10, §5.2 · p. 10

5.3 Cost Scalability

• 24.9× lower per-turn cost than OpenAI on MSP-PODCAST accompanies flat, bounded cost as conversation history grows.The context projector shifts less relevant history to transcripts and summaries; transcription and summarization account for 5.1% of full cost on MSP-PODCAST.Source: Table 2 · p. 10, §5.3 · p. 10, Figure 7 · p. 10

5.4 Context Efficacy

• On RT2, llmovoice retains 98.7% of OpenAI-level answer quality and preserves the best answer quality compared with RAG and Compaction.The transcript-to-summary failure rate is 14.6%, counting cases where the answer is incorrect and the required fact is inferable from the full transcript but not the loaded summary.Source: §5.4 · pp. 10–11
• Page-level retrieval has lower hit rates on multi-page than single-page queries; llmovoice organizes vpages into coherent vthreads for thread-level retrieval of connected evidence.Page-RAG multi-page hit rate drops by 26.4% on RT2 and 27.4% on MSP-PODCAST relative to single-page queries.Source: Figure 10 · p. 11, §5.4 · p. 11

5.5 System Overhead

• 78.4 ms (3.8%) on RT2 and 50.9 ms (2.7%) on MSP-PODCAST are the memory-to-context overheads; turns with a control function call incur 790.2 ms mean decision latency.The latency distributions show earlier, steeper rises and shorter high-latency tails for llmovoice than OpenAI on both workloads.Source: §5.5 · pp. 11–12, Figure 11 · p. 11, Table 4 · p. 12

5.6 Evaluating Llmovoice in the Real World

• In real-world cases, llmovoice retrieves a prior travel constraint at $0.13/min, avoids false-trigger cascades during packet loss, and resumes a thread after reconnection.The travel-planning baseline with context compaction forgot the user’s constraint; the OpenAI Realtime baseline treated reconnection as a new session.Source: §5.6 · p. 12

LLM-over-voice systems

• Speech-native models target speech quality, latency, and duplex interaction, while llmovoice adds persistent serving states for delivery and audio-arrival conditions.The paper positions explicit voice states and context orchestration as addressing a gap in existing model architectures.Source: §6 · p. 13

Persistent conversational memory

• Persistent-memory systems store and recover prior information; llmovoice’s vpage records voice interactions with available representations and retrieval embeddings.Those records preserve semantic content and speech-delivery evidence for later organization, retrieval, and orchestration.Source: §6 · p. 13

RAG and context reduction

• Unlike document-oriented RAG and text-centric reduction, llmovoice organizes temporally ordered voice records and selects audio, transcript, summary, or omission under a budget.Caching optimizes processing of an already constructed context, whereas llmovoice determines which voice context to construct.Source: §6 · p. 13

7 Conclusion

• llmovoice explicitly models semantic, paralinguistic, and environmental states in voice interaction.It is presented as voice-context management middleware.Source: §7 · p. 13
• Its orchestration loop jointly reasons over voice states to derive runtime controls and constructs bounded context from structured interaction units at multiple fidelities.This replaces reliance on monolithic full history.Source: §7 · p. 13
• Across benchmarks and end-to-end cases, llmovoice improves speaking-rate alignment and robustness to network fluctuations while reducing long-session costs and preserving response quality.Source: §7 · p. 13