Beyond Text: Utilizing Vocal Cues to Improve Decision Making in LLMs for Robot Navigation Tasks

1 Introduction

• Beyond Text combines audio transcription with affective vocal features, including pitch, loudness, and duration, to improve navigation under ambiguous or uncertain instructions.The approach aims to use vocal cues to interpret navigational instructions more reliably.Source: §1 · p. 2
• With affective vocal cues added beyond text, the approach achieved a 70.26%+ winning rate for detecting uncertainty and generating appropriate next-step actions; GPT3.5, GPT3, and Gemini Pro all showed significant increases.Source: §1 · p. 2
• DNIA comprises 500 human audio navigational instructions with perceptual ground-truth labels and diverse semantic and vocal ambiguities.The dataset is presented as an experimental foundation for the method and future research.Source: §1 · pp. 2–3
• Under token-manipulation attacks on prompts fed to LLMs, the framework had a 22.44% lower decrease ratio in winning rate than the text-only language model; ablation of three vocal cues increased winning rate by 15% to 20%+.Source: §1 · p. 3

Large Language Models for Robotics

• Beyond Text assesses human uncertainty in potentially unreliable audio navigation instructions, unlike Ren et al. (2023), which relies on human selection among generated options.Prior robotics work includes LLM-based semantic prediction for navigation and multimodal long-horizon instruction generation.Source: §2 · p. 3

Foundation Models with Audio

• Audio transcription advances foundation models, but existing research does not specifically address audio inputs for human-robot navigation or extracting vocal uncertainty from speech.The cited work also explores speech generation, audio-visual understanding, and direct audio processing for speech translation and spoken queries.Source: §2 · pp. 3–4

Uncertainty Modeling in LLMs

• Prior LLM uncertainty work includes comparing black-box uncertainty measures and using entropy; Beyond Text defines confidence through divergence from human-perception distributions.The confidence score is inversely proportional to the Kullback-Leibler divergence between the LLM prediction and ground-truth human perception distribution.Source: §2 · p. 4

Human Affective Analysis in Robotics

• Beyond Text analyzes both speech content and vocal tones to identify emotions and ambiguities in social-navigation commands.The related-work discussion notes that RxR does not cover affective cues in vocal instructions.Source: §2 · p. 4

3 Our Approach

• The framework maps audio to a transcript and an affective cue set, combines them in the LLM prompt, and uses the model to select a candidate next step.The candidate actions are generated using few-shot prompts.Source: §3 · p. 4

3.1 Semantic Uncertainty Quantification

• Transcribed speech is assessed for textual uncertainty, self-corrections, and hesitation signals, which prompt the LLM to adjust confidence in subsequent instructions.Examples include hedging phrases, speech repairs, and pauses.Source: §3.1 · pp. 4–5

3.2 Vocal Affective Cue Analysis

• Duration, pitch, and loudness changes are treated as possible vocal uncertainty cues and aligned with instruction segments.The method measures unusually long or short sub-instructions and uses timestamps to align pitch and loudness changes; it excludes the first and last three seconds to avoid recording noise.Source: §3.2 · p. 5

3.3 In-Context Prompting

• Few-shot examples cover language-only, vocal-only, and combined uncertainty, while the LLM proposes five actions ranging from following the instruction to asking another person.The first option blindly follows the transcript; the final option is a fail-safe request for directions from someone nearby.Source: §3.3 · p. 6

3.4 Scoring Choices by Polling LLMs

• To reduce sequence-length bias in scoring generated plans, the method selects the most probable label from A–E and compares confidence with and without vocal prompts.The confidence change is defined as ΔC = C(ρvocal) − C(ρtext).Source: §3.4 · p. 6

Dataset composition

• DNIA contains 500 navigation audio clips from 20 people, with ten men and ten women; clips are 10–30 seconds and include native and non-native vocal styles.The recordings come from volunteers giving directions to locations inside buildings they had visited.Source: §4 · p. 6

Figure 4

• Figure 4 reports top log-probability choices by context and audio type; Beyond Text shifts choices away from A toward B–E compared with transcription-only.The caption characterizes B–E as identifying uncertainty in the audio.Source: Figure 4 · p. 7

Uncertainty categories

• The dataset labels 285 clips as language uncertainty and 215 as vocal-tone uncertainty, where the transcript alone does not indicate uncertainty.The authors report written consent, anonymization, and review of recordings for personal information.Source: §4 · p. 7

Human evaluation

• Human evaluation uses five answer options, with 20 separate participants asked to listen and choose within one minute per clip.The participants were recruited with diverse genders and dialect backgrounds to assess uncertainty in audio-guided instructions.Source: §4 · p. 7

5.1 Evaluation Metric: Confidence measure

• The confidence measure decreases as the KL divergence between the model’s candidate-response distribution and the ground-truth distribution increases.The model’s distribution assigns probabilities to possible candidate responses; the paper seeks to reduce KL divergence and raise confidence.Source: §5.1 · p. 8

5.2 Confidence Score Improvement

• Beyond Text significantly surpassed transcription-only, text-only, and Chain-of-Thought comparisons in confidence score, with affective cues lowering variance across two categories.The analysis uses a subset of 100 instances aligned with user selections and average confidence across all 500 clips.Source: §5.2 · p. 8

5.3 Choice Distribution Change

• With transcription alone, choice (A) was top-log-probability for 220 of 500 clips; with affective audio augmentation, that count fell to 15.Choice (A) reflects following the instruction without identifying uncertainty; the desired alternatives identify uncertainty and guide an appropriate next step.Source: §5.3 · pp. 8–9

5.4 Winning Rate

• 70.46% overall and 72.56% for VU were the winning rates reported for Beyond Text, which outperformed transcription-only and reasoning-augmented approaches.The paper reports these figures for the overall dataset and VU interpretation, respectively.Source: §5.4 · pp. 9–10, Table 3 · p. 9

5.5 Ablation Study

• Without reasoning, loudness alone produced the highest winning rate among vocal cues; with reasoning, vocal cues strengthened uncertainty interpretation and navigation decisions.The authors suggest that multiple vocal features without reasoning may overwhelm the model and hinder its ability to connect vocal characteristics with uncertainty.Source: Table 2 · p. 9, §5.5 · p. 10
• Beyond Text improved winning rates on all three tested LLMs: GPT3.5, GPT 3, and Gemini.The paper points to Table 4 for the model-specific results.Source: §5.5 · p. 10, Table 4 · p. 10

6 Robustness to Adversarial LLM Attacks

• Token manipulation replaces uncertainty-related wording with deterministic language before prompting the LLM to reproduce the altered instruction.The attack pipeline is applied after transcription and adjusts the prompt to match the paraphrase.Source: §6 · p. 11
• 33.43% for Beyond Text versus 55.87% for transcription-only: the reported performance decreases under token manipulation; Table 5 presents the comparative robustness results.The attack paraphrases the transcript to replace uncertainty semantics with deterministic language and aligns the prompt to the altered transcript.Source: §6 · p. 11, Table 5 · p. 11

7 Conclusions and Open Questions

• Beyond Text integrates human-voice affective cues and, in a user study, outperforms text-based techniques in win rate and entropy confidence.The conclusion presents DNIA as a dataset developed to support research in this direction.Source: §7 · p. 11
• Vocal cues are intended to help robots infer uncertainty without repeated clarification, which may be infeasible when real-time interaction is important.The authors describe this as complementary to approaches that reask humans, and note that disfluencies can also arise from distractions.Source: §7 · pp. 11–12
• The authors call for more audio features, including intonation and prosody, and for clearer accounts of how vocal features relate across tasks.They also describe DNIA as foundational and plan to expand it with additional vocal features.Source: §7 · p. 12
• A corrected command can resemble uncertainty, as in a speaker who changes a left turn to a right turn; the authors identify fine-grained uncertainty definition as unresolved.They describe Beyond Text as a first step and call for future work on this problem.Source: §7 · p. 12
• Direct audio processing remains an open direction: cited work reports success on speech tasks, but the models still lack full understanding of emotional nuances such as uncertainty.The passage refers to Tang et al. (2023).Source: §7 · p. 12
• Real-world social-robot deployment faces the added complexity of multiple audio sources beyond human speech.The authors identify these sources as an issue needing research attention.Source: §7 · p. 12

Broader Impact Statement

• Vocal-cue interpretation may be biased by gender, accent, or cultural background, and the approach is limited by current LLM capabilities.The authors recommend thoughtful use, particularly in less sensitive situations, to reduce downstream bias risks.Source: Broader Impact Statement · p. 12

A LLM Implementation Design

• GPT-4 generates choices and reasoning, then GPT-3.5 scores confidence; JSON outputs support consistent parsing, and parallel API processing speeds the workflow.The implementation specifies GPT-4 for choice generation and GPT-3.5 for confidence scoring.Source: A LLM Implementation Design · p. 19

B Transcription Model Prompting

• Whisper-Small transcribes the audio, with a custom prompt intended to preserve mumbling, hesitations, and speech repairs as uncertainty-related signals.The model is described as having 244M parameters, requiring 2GB VRAM, and transcribing within one second.Source: B Transcription Model Prompting · p. 19

C.1 Prompts and Examples for Generating Reasoning and Choices (GPT-4)

• The GPT-4 prompt requests five distinct next-step actions and directs the model to seek further instruction where uncertainty appears.The examples contrast prompts without an affective cue with prompts that include one.Source: §C.1 · p. 20, §C.1.1 · p. 20, §C.1.2 · p. 22

C.2 Prompts and Examples for Choice Selection (GPT-3.5)

• GPT-3.5 selection prompts vary between transcript-only, transcript plus reasoning, and transcript plus reasoning and affective cue.In the vocal-uncertainty example, the model changes from choice A without Beyond Text to choice B with it.Source: §C.2 · p. 23, §C.2.1 · p. 24, §C.2.2 · p. 25

C.3 Large Language Model Adversarial Token Manipulation Attack Prompt

• The attack prompt uses GPT-4 to remove uncertain wording and add certainty terms while preserving the timestamped-transcript format.The prompt describes changing the certainty level of an instruction from an audio transcript.Source: §C.3 · p. 27

D Adversarial Attack Example

• The first transcript says the speaker will “probably need” to turn left, while the altered version says the speaker “will make” a left turn.The compared instructions otherwise describe the route from Lab A past the locker room toward Lab B.Source: D Adversarial Attack Example · p. 28
• Both versions report no loudness change and a large pitch change at 23.000 seconds.The annotations list the same segment durations for both transcripts.Source: D Adversarial Attack Example · p. 28
• In both versions, the left-turn phrase spans 12.440–19.840 seconds, a duration of 7.40 seconds.The listed segment is the instruction whose certainty wording differs between the transcripts.Source: D Adversarial Attack Example · p. 28

E DNIA Dataset Details and Samples

• For VU clips, dataset construction excludes semantic uncertainty signals (uncertainty-related words, speech repairs, hesitations), while duration, pitch, and loudness are affective cues associated with speech uncertainty levels in robot navigation tasks.Source: E DNIA Dataset Details and Samples · p. 28
• The dataset is available with grounded textual annotations and Whisper-generated transcriptions for researchers seeking to reproduce the results.Source: E DNIA Dataset Details and Samples · p. 29
• The 500-clip study asked 20 participants from varied gender and dialect backgrounds to listen carefully and choose within one minute per clip.Source: E DNIA Dataset Details and Samples · p. 29
• Most correct choices fell among options B, C, D, and E; the passage says this aligns with Figure 4 results showing higher accuracy than baselines.Table 7 presents the human-choice distribution.Source: E DNIA Dataset Details and Samples · p. 29, Table 7 · p. 29

F Pitch and Loudness Shifts Detection

• Algorithm 1 samples pitch and loudness each second, flags threshold-exceeding changes, then returns timestamps of the largest shifts for aligning vocal-cue prompts with instructions.Source: F Pitch and Loudness Shifts Detection · p. 29

• Beyond Text consistently outperformed baseline methods in winning rate across loudness thresholds of 0.03 and 0.05 and pitch thresholds of 300 Hz and 500 Hz.The Transcription Only LLM scored 22.16% total, 21.75% LU, and 22.79% VU; With Chain-of-Thought Reasoning scored 49.30% total, 58.60% LU, and 36.74% VU.Source: G Sensitivity Study to Loudness and… · pp. 29–30, Table 8 · p. 30

• Users listen to a randomly selected audio clip, choose the most accurate and efficient of five answers, then confirm to submit and continue.Source: H User Interface for Human Evaluation… · p. 30, Figure 7 · p. 30