PaperlayerView exact revision on arXiv

Source-linked AI summary

VoiceLongMemEval: Do Assistants Remember How You Sounded?

Ramit Pahwa, Parivesh Priye, Apoorva Beedu

arXiv:2609.00570v1cs.AIcs.LG

TL;DR

Existing long-term memory benchmarks do not test whether assistants retain paralinguistic delivery across sessions, even though delivery can determine a user’s meaning. VoiceLongMemEval addresses this gap with adversarially gated questions whose answers depend on annotated or spoken delivery, finding consistent affect gaps and a deficit in cascaded ASR pipelines. The benchmark also reports author-supported scope limits involving synthetic data, oracle-length contexts, audio-model coverage, generated questions, judging bias, and prompting-dependent certification.

  • Problem

    Long-term conversational memory and paralinguistic perception have been measured separately, leaving assistants’ memory for how something was said across sessions untested.

  • Method

    VoiceLongMemEval combines 523 questions, delivery annotations, multiple speech renders, and adversarial gates that ensure answers cannot be recovered from words alone.

  • Results

    +0.09 to +0.38 affect gaps occur across eight systems, while audio-native models reach 0.354–0.412 versus 0.325 blind and cascaded systems underperform.

  • Takeaways & Limitations

    Paralinguistic metadata remains indispensable for adversarially gated items, and audio-native models outperform cascaded frontier models on identical clips.

  • Takeaways & Limitations

    The study uses synthetically authored items, oracle-regime contexts of ∼10–15k tokens, only two 7B audio models, LLM-generated questions, and LLM judging.

Abstract

from arXiv · show

With the growing scale of multi-agent architectures and large language models, deployed AI assistants are increasingly tasked with reasoning over long, continuous, multi-session conversation histories. Current benchmarks evaluate this dialogue history as information retrieval over long horizon, temporal reasoning, or knowledge updates, while crucially ignoring the fundamental dynamics of human-agent interaction, i.e. how they said it. To address this gap, we present VoiceLongMemEval (VLME) benchmark, where every answer depends on paralinguistic metadata (emotion labels, prosody descriptors, and voice events) attached to conversational turns, which is otherwise unrecoverable from the words alone. Every item passes a three-stage adversarial gate, ensuring that a strong language model fails when given only the transcript. Evaluating leading frontier and open-weight models reveals a pervasive affect gap; providing text-track paralinguistic metadata yields a 0.09 to 0.38 accuracy boost (0.61 to 0.69 when prompted with evidence hints), while standard ASR pipelines systematically discard this signal. Additionally, audio-native models successfully extract these cues directly from speech (0.354 to 0.412 vs. 0.325 blind). Code and dataset will be made available upon acceptance.

1 Introduction

VoiceLongMemEval targets a missing intersection between long-term conversational memory and paralinguistic perception: remembering how something was said across sessions. It introduces a benchmark whose answers depend on delivery metadata rather than words alone.

  • Motivation: Long-term memory and paralinguistic perception have been evaluated separately, leaving cross-session memory for how something was said untested.The missing capability includes retaining delivery, updating delivery shifts, and retrieving them later for a specific question.
  • Benchmark: VoiceLongMemEval embeds paralinguistic needles in conversational sessions so answers are recoverable only from delivery metadata.A three-part adversarial gate enforces that the words alone cannot provide the answer.
  • Benchmark: 523 adversarially validated items span six paralinguistic-memory question types, each with abstention variants.The types cover affect recall, affective preference, affect update, cross-session affect, temporal-affective reasoning, and prosody-disambiguated interpretation.
  • Findings: +0.09 to +0.38 affect gaps appear across eight evaluated systems, with emotion tags providing the strongest signal.Ablations also report that chain-of-thought reasoning cannot substitute for missing context.

2 Related Work

Prior work studies long-term conversational memory and paralinguistic understanding as separate capabilities. VoiceLongMemEval connects them by testing whether vocal cues persist in memory and remain usable for later questions.

  • Long-term conversational memory: Long-term memory benchmarks evaluate extraction, multi-session reasoning, temporal reasoning, knowledge updates, abstention, personalization, and social interaction recall.The cited benchmarks include MSC, LoCoMo, LongMemEval, LongMemEval-V2, PerLTQA, MemBench, and DialSim.
  • Paralinguistic understanding in speech-LLMs: Paralinguistic benchmarks measure emotion, prosody, speaker attributes, and context-sensitive responses in audio-language systems.Examples include Dynamic-SUPERB, AIR-Bench, Audio2Tool, AudioBench, MMAU, MMSU, SD-Eval, and CP-Bench.
  • Cascaded versus audio-native pipelines: Most production voice assistants remain cascaded ASR-to-LLM systems, whose transcription boundary discards prosody before long-term memory retrieval.Prior cascade-versus-native comparisons were confined to single-turn understanding, whereas VLME measures the loss at memory level.

3 Benchmark Construction

VLME extends long-memory records with structured delivery annotations, adversarially validates that items cannot be solved from transcripts, and constructs multiple question families and speech renders. Its corpus combines controlled evidence sessions with distractor histories and synthetic audio.

  • Corpus: VLME contains a 202-question adversarially gated core and two derived families totaling 523 questions over paralinguistically annotated evidence sessions.The benchmark is designed around an invariant: answers are recoverable from the paralinguistic channel but not the words alone.
  • Paralinguistic layer: Each user turn receives emotion, prosody, voice-event, pragmatic, and acoustic-only delivery annotations.Emotion uses 12 everyday labels; prosody covers rate, pitch, loudness, pauses, and emphasized words, while voice events include laughs, sighs, coughs, throat-clears, and gasps.
  • Paralinguistic layer: The benchmark provides blind transcript-only, descriptive metadata, and audio renders, with the blind render serving as the control and adversary view.The descriptive render includes acoustic stage directions and a structured tagged variant.
  • Session construction: Evidence sessions enforce lexical flatness, affect against pragmatics, question neutrality, and annotation uniformity to prevent textual or annotation-presence shortcuts.A deterministic assembler adds topic-screened filler and emotive distractors, scales distractors with evidence-session count, and stratifies needle positions over time.
  • Adversarial validity gates: 0 of 175 non-abstention taxonomy items were blind-solvable after terminal gating, while the 72B aware-solve rate was 57.9%.Derived families rely on gated evidence sessions and mechanical invariant checks rather than separate blind attacks.
  • Question generation: Six taxonomy types isolate affect recall, preference, update, cross-session aggregation, temporal-affective reasoning, and prosody-disambiguated interpretation.Twenty-seven abstention items presuppose an emotional episode that never occurred and penalize affect hallucination.
  • Audio synthesis: The audio clips are two-speaker synthetic sessions generated with Dia and emotion-matched RAVDESS reference clips.The audio evaluation uses machine-generated speech, and human listening passed 91 of 104 clips.
  • Audio synthesis: The initial Whisper-large-v3 speech-emotion gate reached only ∼30% on acted RAVDESS and was moved to an advisory check.The authors associate this with cross-corpus speech-emotion-recognition bias.

4 Experimental Setup

The experiments evaluate eight proprietary and open-weight language models on long conversational contexts with distractor sessions. Results are averaged across three seeds and tested with paired statistical procedures.

  • Models: Eight models comprise three proprietary frontier systems and five open-weight systems.The proprietary models are Claude Opus 4.8, Claude Sonnet 4.6, and GPT-5.5; open-weight systems include Llama, Qwen, and Gemma models.
  • Evaluation protocol: Each test case places target evidence among five randomly sampled distractor sessions in roughly 10k–15k-token contexts.Models generate free-text answers judged against task-specific rubrics, with distractor selection and arrangement controlled across seeds.
  • Evaluation protocol: Three question-set conditions contain 181 nuanced, 202 original, and 140 indirect questions, with all reported gaps significant at p < 0.001.Pairwise comparisons use 10,000-iteration paired bootstrap resampling and McNemar’s test.

5 Results

Across models, paralinguistic metadata produces a consistent affect gap on adversarially validated long-memory tasks, while cue format, question phrasing, prompting, and audio processing strongly influence performance.

  • 5.1 The Affect Gap on Original 202Q: Affect gaps range from +0.089 for Gemma 3-12B to +0.383 for Opus 4.8, remaining positive across proprietary and open-weight models.Blind accuracy stays uniformly low at 0.09–0.18, and headroom normalization preserves the model ranking.
  • 5.1 The Affect Gap on Original 202Q: Affect gaps are largest when delivery directly determines the answer and smallest when reasoning must integrate evidence across sessions.The same type-level ordering appears across both evaluated models.
  • 5.2 What Metadata Component Matters?: Wrong metadata barely exceeds blind accuracy, indicating that models use annotation content rather than merely benefiting from metadata presence.Wrong-metadata scores are 0.228 for Opus and 0.183 for GPT, versus blind scores of 0.193 and 0.129.
  • 5.2 What Metadata Component Matters?: Emotion-only input outperforms full descriptive metadata, with scores of 0.614 for Opus and 0.535 for GPT.The results suggest categorical emotion labels are easier to integrate than free-text acoustic descriptions.
  • 5.2 What Metadata Component Matters?: Tagged metadata exceeds descriptive text by +0.168 for Opus and +0.149 for GPT, while prosody-only and events-only provide partial signal.The condition ranking is identical across both models, suggesting model-independent cue informativeness.
  • 5.4 Can Prompting Fix the Indirect Gap?: Question phrasing changes the gap from +0.61 to +0.69 for nuanced questions to +0.11 to +0.18 for indirect natural questions.A retrieval-time hint improves natural-question performance, but adversarially gated 202Q remains robust: blind accuracy rises only +0.079 for Opus.
  • 5.4 Can Prompting Fix the Indirect Gap?: Increasing distractors from 3 to 10 mildly reduces descriptive accuracy while leaving blind accuracy flat, so the affect gap persists across haystack sizes.This pattern holds for frontier and open-weight models.
  • 5.4 Can Prompting Fix the Indirect Gap?: Audio-native models score 0.354–0.412, whereas Whisper-to-Opus scores 0.254 because transcription strips delivery cues and introduces content errors.A hint raises Opus cascade performance from 0.254 to 0.515 (+0.261).

6 Discussion and Limitations

The discussion recommends preserving structured paralinguistic metadata and explicitly eliciting affective reasoning, while identifying synthetic data, limited context lengths, audio models, generated questions, judge bias, and prompting-dependent gates as limitations.

  • Implications for memory system design: Structured paralinguistic metadata should be retained, and retrieval should explicitly elicit affective reasoning.For Claude Opus, tagged metadata outperforms descriptive natural-language metadata, 0.757 versus 0.589.
  • Implications for memory system design: 0.254 Whisper →Opus performance trails 0.354–0.412 for 7B audio-native models on identical clips.The comparison frames the cascade shortfall as an architectural issue rather than a capability ceiling.
  • Implications for memory system design: +0.067 controlled metadata contribution shows that prompting and annotation provide distinct, complementary benefits on generated items.For adversarially-gated items, metadata remains indispensable even with strong prompting.
  • Limitations: The evaluation is limited by synthetically authored items, an untested ∼100k-token regime, only two 7B audio models, generated-question biases, and possible LLM-judge bias.The paper also notes that human evaluation would strengthen results.
  • Limitations: Prompting-dependent gate certification limits the interpretation of adversarial validation because prompted frontier models reach 0.267 on 202Q blind items.The gates certify items against an unprompted 72B adversary.

7 Conclusion

VoiceLongMemEval finds that paralinguistic metadata improves conversational memory across models and conditions, while audio-native models outperform cascaded systems on identical clips. The benchmark supports prompting and annotation as practical interventions and is released for further research.

  • Conclusion: Paralinguistic metadata improves conversational memory across eight models, question types, distractor scales, and audio modalities.The affect gap persists across these evaluation dimensions.
  • Conclusion: A retrieval-time hint recovers much of the signal, while metadata adds +0.067 controlled contribution.Prompting and annotation are described as partially interchangeable and complementary.
  • Conclusion: Audio-native 7B models outperform cascaded frontier models on identical clips, quantifying the ASR pipeline’s paralinguistic deficit.The comparison concerns memory-level use of vocal cues.
  • Conclusion: The released benchmark is intended to support research on the paralinguistic dimension of long-term memory.The paper states that the benchmark will be made available.

A Error Analysis

The error analysis separates benchmark items by whether metadata changes outcomes and defines six question types, including abstention cases that test whether models hallucinate emotional episodes. The standard and hint prompts differ only by retrieval-time guidance about delivery cues.

  • Error Analysis: The analysis categorizes 202 items by the joint blind and descriptive outcomes for Claude Opus 4.8.The categorization is reported in Table 9.
  • Error Analysis: 42% of items are gap contributors where paralinguistic metadata changes success to failure, while 78 hard-for-both items remain unsolved with metadata.The 29 easy/lexical items retain residual text signal despite adversarial gates.
  • Error Analysis: Six question types factor paralinguistic-memory competence, each with an abstention variant comprising 15% of items.Abstention questions presuppose an emotional episode that never occurred, making “never expressed” the gold answer.
  • Error Analysis: The benchmark’s six types are presented in Figure 1 and Table 10, comparing what words alone suggest with what delivery reveals.The gap between those readings is the measured benchmark target.
  • Prompt conditions: The standard text prompt supplies conversation history and asks for a concise answer based on it, with abstention when information is insufficient.The prompt includes the question date and the user’s question.
  • Prompt conditions: The hint prompt prepends guidance to consider tone, pace, volume, pauses, laughter, and sighs as indicators of feelings, intentions, and preferences.The audio prompt likewise asks the model to listen to the conversation and answer concisely.

D Qualitative Examples: Hint vs. No-Hint

Qualitative examples show that retrieval-time hints redirect models from literal lexical interpretations toward affective interpretations grounded in delivery cues. Slow, flat, hesitant, or sighing speech supports readings such as relief, challenge, obligation, and uncertainty rather than celebration or enthusiasm.

  • Experimental presentation: The examples compare standard and hint-prompt answers for Claude Opus 4.8 under a descriptive condition with three distractors.Each example includes an evidence session with paralinguistic metadata while omitting distractor sessions for space.
  • Example 1: coverage belief: In the coverage example, slow, quiet delivery with hesitations and sighs indicates that accepting the arrangement feels challenging despite agreeable words.The hint shifts interpretation from contained logistics to reservations about the burden.
  • Example 2: yoga belief: In the yoga example, slow, flat, quiet delivery with long pauses suggests obligation rather than enthusiasm, making full session use unlikely.The no-hint response cannot predict usage from the purchase and schedule alone.
  • Example: certification news: In the certification example, slow, flat speech with a pause and sighs supports relief rather than celebration.Without the hint, the model treats passing the exam as celebratory; with it, the model answers “Probably not” about updating the credentials file.

2. Limitations

The paper reports limitations concerning its experimental scope and methodological assumptions, while noting that it is an empirical study without theoretical results.

  • The paper has a dedicated Limitations section covering the evaluated context-length regime, LLM-generated sessions, TTS-rendered audio, derived question families, and prompting sensitivity.
  • The study is empirical and contains no theoretical results requiring assumptions or formal proofs.

4. Experimental result reproducibility

The paper describes its evaluation conditions and provides reproducibility materials, including an anonymized repository with benchmark assets and evaluation code.

  • The paper describes reproducibility expectations for architectures, models, datasets, and closed-source systems, including access paths or construction instructions.
  • An anonymized repository provides the benchmark, prompts, gating rubrics, audio-synthesis scripts, manifests, evaluation code, and reproduction instructions.The de-anonymized version is planned for public release upon publication.
  • The paper specifies evaluation conditions, context regimes, models, decoding settings, random seeds, and appendix-level per-condition details.No models are trained, so optimizer and training hyperparameters do not apply.

7. Experiment statistical significance

The supplied material documents statistical reporting, compute and ethics disclosures, societal impacts, safeguards, asset attribution, documentation, and the absence of human-subject experiments.

  • Main results are reported as mean ± standard deviation over three evaluation seeds with fixed items.The variability captures decoding and judging nondeterminism.
  • The experiments are inference-only, using hosted APIs for frontier models and a single GPU node for open-weight models, with approximate evaluation costs reported.
  • The paper states that synthetic sessions and TTS-rendered audio involve no human subjects or personal data and conform to the NeurIPS Code of Ethics.
  • The paper discusses benefits of affect-aware memory systems alongside privacy and profiling risks from persistent affective metadata storage.User control over stored metadata is noted as a mitigation.
  • The benchmark introduces no models and uses synthetic text sessions and TTS-rendered audio, which the paper characterizes as posing no high misuse risk.
  • Existing assets are cited and used under their respective licenses and terms, while the new benchmark is documented in the paper, appendix, and repository.
  • The study uses neither crowdsourcing nor human-subject research; sessions are LLM-generated and audits use automated adversary–judge gates and static checks.
Loading 2609.00570v1…