Source-linked AI summary
Not All Fallbacks Are Failures: Understanding and Recovering from Fallbacks in Mobile Voice Assistants
Phillip Schneider, Alexandre Mercier, Joshua Oehms, Kristiina Jokinen, Florian Matthes
TL;DR
Deployed voice assistants encounter heterogeneous fallback situations that generic responses do not resolve well. This paper constructs and annotates real-world fallback data, evaluates taxonomy-aligned routing models, and finds that lightweight embedding classifiers generally outperform LLMs with lower latency and computational cost.
Problem
Generic fallback handling does not distinguish heterogeneous causes such as accidental activations, ambiguous requests, and unsupported capabilities, limiting interaction recovery.
Method
The paper builds the VOXFALLBACKS dataset and evaluates a taxonomy-aligned two-stage classification pipeline using embedding-based classifiers and instruction-tuned LLMs.
Results
Lightweight embedding classifiers consistently outperform LLMs for first-stage routing, with E5 + LR achieving F1 = 0.852 and classifier latency below 14 ms versus 184–1,661 ms for LLMs.
Takeaways & Limitations
Fallback handling is best treated as structured recovery, prioritizing low-latency discriminative routing and reserving generative models for targeted clarification.
Takeaways & Limitations
The dataset contains only single-turn utterances, and the evaluation does not measure whether generated clarifications resolve user goals through end-to-end task success.
Abstract
from arXiv · showhide
Robust understanding of user input is a core requirement for voice assistants deployed in real-world environments. In practice, these systems encounter heterogeneous fallback situations caused by noisy audio input, transcription errors, ambiguous requests, incomplete utterances, or unintended activations. Existing systems typically respond with generic fallback messages, which do not resolve the underlying interaction failure and can degrade user experience. We study fallback handling in a deployed smartwatch-based voice assistant for general health support in everyday environments. Our analysis is based on six months of real-world usage data from more than 500 users, yielding a dataset of 3,030 anonymized, naturally occurring fallback-triggering utterances. We contribute (1) an operational taxonomy and the annotated VoxFallbacks dataset of these interactions, (2) a comparative evaluation of different models within a classification pipeline under practical deployment constraints, and (3) practical lessons for designing robust and cost-efficient fallback mechanisms. Results show that lightweight embedding-based classifiers outperform larger generative models on most classification tasks while requiring substantially fewer computational resources.
1 Introduction
Deployed voice assistants face heterogeneous fallback-triggering inputs that generic responses do not adequately address. This paper frames fallback handling as structured recovery and evaluates taxonomy-driven routing under deployment constraints.
- Production fallbacks arise from noisy environments, recognition errors, incomplete utterances, unsupported requests, and accidental activations.
- Generic fallback responses often fail to distinguish underlying causes, limiting support for interaction recovery.
- Accidental activations require suppression because intrusive responses can increase user frustration and contribute to churn.
- The paper unifies out-of-domain detection, clarification, and accidental-activation filtering as fallback recovery under latency, cost, and maintainability constraints.
- The study contributes an operational taxonomy, the VOXFALLBACKS dataset with 3,030 utterances, comparative model evaluation, and deployment-oriented design insights.
- Lightweight embedding classifiers outperform instruction-tuned LLMs for fallback routing, while targeted LLM use can preserve efficiency requirements.
2 Related Work
Prior research treats accidental activation filtering, conversational breakdown recovery, and ambiguity clarification as largely separate areas. This work positions fallback handling as their combined problem within real-world voice-assistant interactions.
- Fallback handling comprises accidental activations, out-of-domain requests and conversational breakdown, and ambiguous utterances requiring clarification.
- Accidental Activations & Unintended Speech: Accidental-activation research focuses on filtering wake-word false positives or linguistic breakdown signals, often separately from broader conversation flow.
- Out-of-Domain Requests & Conversational Breakdown: Conversational breakdown research studies re-asking, apologizing, capability disclosure, collaborative repair, and distinctions between recovery initiators.
- Ambiguous Utterances & Clarification: Clarification research reports that LLMs may silently assume intent or produce over-generic probes instead of meaningful clarification.
- Ambiguous Utterances & Clarification: The paper empirically supports decoupling robust classification with lightweight models from contextually targeted clarification generation with LLMs.
3 Dataset Construction
The dataset is drawn from a deployed smartwatch assistant serving older and chronically ill users, then cleaned, anonymized, deduplicated, and hierarchically annotated. The resulting corpus emphasizes diverse fallback phenomena rather than raw production frequency.
- The smartwatch assistant supports emergency routing, health tracking, reminders, microdialogues, and informational lookups for elderly and chronically ill users in the DACH region.
- Audio and speech-to-text capture begin through either manual smartwatch activation or proactive assistant-initiated turns.
- Fallbacks were collected over six months from more than 500 predominantly 60–90-year-old individuals, comprising approximately 6–7% of assistant requests.
- The preprocessing retained dialogue-initiating utterances, removed duplicates and empty transcriptions, anonymized personal information, and randomized utterances.
- Deduplication targeted diversity rather than operational frequency; an SVM yielded weighted F1 = 0.87 and Jensen–Shannon distance = 0.119 between distributions.
- The taxonomy is a three-level decision tree separating user intentionality, intent clarity, and capability availability.
- Four German-speaking researchers annotated 3,030 utterances, with Fleiss’ Kappa κ = 0.759 on a shared 350-utterance validation subset.
4 Experimental Setup
The evaluation aligns a two-stage fallback-routing pipeline with the taxonomy and compares embedding classifiers against instruction-tuned LLMs. Models are assessed with stratified cross-validation and deployment-relevant performance and latency measures.
- Stage 1 distinguishes intended from unintended utterances, suppressing unintended inputs; Stage 2 routes intended inputs by clarity and capability availability.
- Embedding models use multilingual-e5-large or BGE-M3 paired with logistic regression, SVM, or random forest, while instruction-tuned LLMs provide a generative comparison.
- Prompts define roles, target and fallback labels, strict output formats, and ambiguity guardrails; retrieval conditioning restricts LLM predictions to five upstream-retrieved intent candidates.
- Retrieval conditioning is evaluated only within the complete end-to-end classification pipeline, so results isolate conditioning effects rather than DIET retrieval quality.
- Evaluation uses stratified 5-fold cross-validation, with Stage 1 metrics including accuracy, precision, recall, F1, ROC AUC, and average precision.
- Stage 2 reports macro-averaged precision, recall, and F1 to account for class imbalance.
- Embedding classifier latency excludes embedding computation, which takes approximately 40 ms per utterance.
5 Results & Discussion
Real-world fallback utterances are highly heterogeneous, spanning unintended input, recoverable requests, semantic ambiguity, and failures across multiple pipeline layers. Across binary gating and multi-class routing, lightweight embedding classifiers generally provide stronger performance with much lower latency than LLMs, while retrieval conditioning has model-dependent effects.
- Dataset Findings: 70.56% of unique fallback utterances are unintended activations, averaging 48.28 characters (SD = 45.50) versus 31.31 for intended clear-intent inputs.The analysis emphasizes diversity of unique utterances rather than absolute operational frequency.
- Dataset Findings: 24.13% of unique fallback utterances are clear-intent requests targeting available skills, concentrated in open-ended domains such as knowledge searches, creative writing, and news lookups.Semantic ambiguities account for less than 2% combined, while structured utility tasks are less frequent within this recoverable subset.
- Dataset Findings: Fallback triggers span physical, speech-to-text, natural-language-understanding, and context layers, including acoustic variation, phonetic transcription errors, self-corrections, out-of-scope requests, and side-speech.These distinct failure sources explain why generic fallback responses are insufficient for real-world recovery.
- Multi-class Classification: BGE-M3 + LR achieves the best multi-class result (macro F1 = 0.810, accuracy = 0.840), exceeding Gemma 4 (macro F1 = 0.727) with classifier latency below 0.05 ms versus 216–877 ms.Embedding computation adds approximately 40 ms per utterance, and the multi-class task covers 18 skills plus two fallback labels, with unclear labels combined.
- Retrieval Conditioning: Retrieval-conditioned prompting is model-dependent: it lowers Gemma 4 macro F1 from 0.727 to 0.675 and GPT-4.1 nano from 0.721 to 0.674, but raises Qwen 3.5 from 0.679 to 0.707.The added retrieval step can alter the precision–recall trade-off and substantially increase latency; retrieval quality was not evaluated independently.
6 Conclusion
The paper frames fallback handling as structured recovery and introduces an annotated dataset of deployed voice-assistant fallbacks. Across two-stage routing, lightweight embedding classifiers outperform LLMs at substantially lower latency, while generative models remain promising for clarification.
- 3,030 annotated fallback utterances underpin the VOXFALLBACKS dataset and a three-level taxonomy for structured recovery.
- Lightweight embedding-based classifiers consistently outperform LLMs for intent routing across the two-stage pipeline.
- The proposed hybrid design uses low-latency discriminative models for routing and generative models as a promising option for clarification.
- Fallback recovery must account for real-world latency, cost, and privacy constraints.
Limitations
The study’s evidence is limited by single-turn data, one deployment context, transcript-only researcher annotations, and a restricted modeling and evaluation scope.
- Single-turn utterances restrict analysis to isolated fragments and omit multi-turn recovery dynamics.
- Data from one system, application context, language, and user population make the taxonomy system-specific rather than universal.
- Transcript-only annotations made by researchers without full conversation history provide guesses about user intent rather than ground truth.
- Evaluation covers lightweight small LLMs and embedding classifiers, not larger models, and does not measure end-to-end clarification success.
- Performance evaluation treats error types uniformly despite asymmetric costs for safety-critical errors.
Ethics Statement
The study reports compliance with data-protection and research-ethics standards. Privacy was addressed through anonymization and multi-phase manual review, while deployment risks in assistive settings were acknowledged.
- All study data were collected and processed in accordance with GDPR regulations.
- The publication dataset was anonymized, and three annotators independently inspected entries to remove or modify personally identifiable information.
- Experiments used local computational resources, avoiding large-scale external compute infrastructure.
- The authors identify misclassification and inappropriate fallback handling as ethical concerns in assistive conversational systems.
A Appendix
The appendix documents the recovery pipeline, annotation agreement, observed failure categories, dataset labels, and classification prompts used in the two stages.
- Pipeline: Figure 3 depicts a hybrid pipeline combining an intent-based dialogue manager with binary and multi-class fallback classification stages.
- Annotation: Table 4 reports inter-annotator agreement for three annotators across 350 items and 21 categories, with κ = 0.7591 ± 0.0532.
- Failure categories: Table 5 presents observed interaction failures across the voice-assistant processing pipeline using real-world dataset examples.
- Dataset labels: Table 6 lists fallback-triggering class labels with short descriptions.
- Prompts: Figures 4–7 show zero-shot, few-shot, and retrieval-augmented prompts for binary and multi-class classification.