PaperlayerView exact revision on arXiv

Source-linked AI summary

VoxMem: Benchmarking Multimodal Memory in Large Audio Language Models

Yang Xiao, Vidhyasaharan Sethu, Eun-Jung Holden, Ting Dang

arXiv:2609.32607v1eess.AScs.SD

TL;DR

Spoken conversational memory requires recovering more than words across prior interactions, yet existing benchmarks cover acoustic evidence, memory operations, and multi-session histories only partially. VoxMem evaluates these dimensions together in a multi-session benchmark across controlled context lengths. Across 15 LALMs, none exceeds 40% accuracy at 32K, and models retain speech semantics better than non-lexical acoustic information.

  • Problem

    Existing spoken-memory benchmarks provide limited, ad hoc coverage of acoustic evidence and memory operations, and insufficiently characterize cross-session memory.

    Source: §1 · p. 1

  • Method

    VoxMem evaluates four acoustic evidence types and four memory operations in multi-session histories, using 3,196 instances across four controlled context budgets from 8K to 64K tokens.

    Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 6

  • Results

    No evaluated LALM exceeds 40% overall accuracy at 32K, and models perform better on speech semantics than on non-lexical acoustic information.

    Source: §1 · p. 2, §4.2 · p. 8

  • Takeaways & Limitations

    The findings suggest that reliable spoken conversational memory requires more than longer audio contexts, and VoxMem provides a taxonomy and benchmark to measure these gaps.

    Source: §1 · p. 2

Abstract

from arXiv · show

Spoken conversational systems must recover information from prior interactions (i.e., memory), yet relevant information in speech extends beyond what was said to who said it, how it was spoken, and what was audible, information that exists only in the audio signal and cannot be recovered from a transcript. Beyond what to remember, memory also demands diverse operations: retrieving a single fact, integrating evidence across turns, tracking an evolving state. Real interactions further unfold across sessions, meaning information accumulates across distinct episodes rather than a single continuous recording. Existing benchmarks fall short on all three dimensions: they focus primarily on lexical content, adopt limited and ad hoc memory operations, and treat memory as a single-session problem. We argue that principled memory evaluation requires jointly characterizing the acoustic evidence to be retained and the operations applied to it, and introduce a taxonomy along these two axes. Building on this taxonomy, we present VoxMem: 3,196 evaluation instances over 34,743 spoken sessions (177 hours) crossing four acoustic evidence types (speech semantics, speaker identity, paralinguistic cues, environmental sound) with four memory operations (information extraction, multi-session reasoning, temporal tracking, and answer refusal), grounded in multi-session histories and stratified across context budgets from 8K to 64K tokens. Evaluating 15 LALMs, no model exceeds 40% at 32K. Models retain what was said far better than who said it, how, or what was audible, a gap that widens for complex operations, grows with history length, and manifests as qualitatively distinct failure modes across evidence types. VoxMem aims to provide a foundation to measure and drive progress on the full scope of spoken conversational memory.

1 Introduction

Spoken conversational memory must recover and use information across growing, multi-session histories, but existing benchmarks cover acoustic evidence, memory operations, and cross-session recall only partially. VoxMem evaluates these dimensions together, and tested LALMs remain far from reliable memory, especially for non-lexical audio information.

1 Introduction

  • Longer histories do not guarantee reliable retrieval and use of information, making evaluation of memory across growing interactions important.Spoken systems must accumulate information across past interactions and ground current responses in that history.Source: §1 · p. 1
  • Existing benchmarks cover acoustic evidence and memory operations partially and ad hoc, while cross-session memory and controlled comparisons across history lengths remain insufficiently characterized.Prior evaluations primarily use isolated recordings or continuous dialogues; comparisons across history lengths can confound length with question difficulty and evidence type.Source: §1 · pp. 1–2
  • VoxMem evaluates spoken memory across four acoustic evidence types and four memory operations in multi-session histories, with 3,196 instances and context budgets from 8K to 64K tokens.Its overview depicts 15 valid combinations in the 4 × 4 taxonomy.Source: §1 · p. 2, Figure 1 · p. 3
  • No evaluated LALM exceeds 40% overall accuracy at 32K, and models retain what was said better than who said it, how, or what was audible.The benchmark also reports qualitatively different error modes across question types.Source: §1 · p. 2

2 Related Work

Prior spoken-memory benchmarks cover acoustic evidence and memory operations unevenly, and mostly evaluate single recordings or continuous dialogues. VoxMem addresses these gaps with a taxonomy spanning both dimensions and multi-session histories.

Acoustic evidence and memory operations

  • Existing benchmarks assess lexical content and selected acoustic cues, but rarely test speaker identity, paralinguistic cues, or environmental sound alongside a broad range of memory operations.This fragmented coverage makes it difficult to distinguish failures by evidence type, operation, or their interaction.Source: §2 · p. 3
  • VoxMem’s taxonomy spans four evidence types and four memory operations across all valid combinations, addressing the fragmented coverage of earlier benchmarks.Earlier evaluations mainly tested memory for what was said, limiting diagnosis of failures across audio evidence and operations.Source: §2 · p. 3

From single-session to multi-session spoken memory

  • Prior benchmarks use long-form single recordings or continuous dialogues, whereas VoxMem evaluates multi-session histories with competing and unrelated sessions.VoxMem holds each question fixed across context lengths to isolate history-length effects.Source: §2 · p. 3

3 The VoxMem Benchmark

VoxMem frames spoken-memory evaluation across acoustic evidence and memory operations, then instantiates that framework in controlled multi-session histories. Its construction fixes question and evidence across four context lengths and validates acoustic necessity and question validity.

3.1 Two-dimensional Taxonomy

  • VoxMem combines four acoustic evidence types with four memory operations to characterize what must be remembered and how it must be used.Evidence types are speech semantics, speaker identity, paralinguistic cues, and environmental sound; operations are information extraction, multi-session reasoning, temporal evolution tracking, and answer refusal.Source: §3.1 · p. 4

3.2 Benchmark Construction

  • The benchmark uses evidence, haystack, and filler sessions to test 15 scenarios, with answer evidence distributed across one session, several sessions, or an ordered sequence.Haystacks resemble the evidence without yielding the answer, while fillers extend histories with unrelated conversations.Source: §3.2 · pp. 4–5
  • Speech synthesis assigns users fixed voices, adds controlled vocal and environmental cues, and supplies user turns as audio while assistant turns remain text.Paralinguistic cues use style controls, and environmental sounds are mixed at 10dB SNR; paired cue-removed renditions support quality control.Source: §3.2 · p. 5
  • Four nested histories—8K, 16K, 32K, and 64K tokens—preserve each question and its evidence while adding haystacks and fillers.Evidence and haystack sessions are distributed throughout histories to prevent position or timestamp from signaling the answer.Source: §3.2 · p. 6

3.3 Quality Control

  • Two-level quality control checks individual sessions and assembled question families for perceptible cues, unique answers, intended operations, and evidence leakage.Question-family checks also test acoustic necessity by replacing audio with transcripts or cue-removed renditions.Source: §3.3 · pp. 6–7
  • 46.2% to 4.4%: transcript-only input reduced audio-native accuracy, while speech-semantics accuracy changed from 75.9% to 71.0%.This check used Gemini-3.7-Flash on all 669 answerable questions at 8K; the table compares transcript-only input with full audio.Source: Table 2 · p. 6, §3.3 · p. 7

4 Evaluation and Analysis

The evaluation measures 15 LALMs across evidence types, memory operations, and history lengths. Models perform best on speech semantics, while acoustic memory remains weaker and access declines as histories grow.

4.1 Experimental Setup

  • The evaluation tests 15 LALMs at 8K, 16K, 32K, and 64K history lengths, measuring length in Whisper-encoder tokens.Free-text responses are judged by an LLM, with GPT-5.6-Luna re-judging yielding κ = 0.95 agreement.Source: §4.1 · pp. 7–8

4.2 Acoustic Evidence

  • At 32K, no evaluated model exceeds 40% overall accuracy, and proprietary models average 33.0% versus 21.9% for open-weight models.The strongest model reaches 38.5%.Source: §4.2 · p. 8
  • At 32K, proprietary models average 55.6% on Speech Semantics, versus 32.7% on Speaker Identity, 20.0% on Paralinguistic Cues, and 21.9% on Environmental Sound.Open-weight means follow the same qualitative separation: 31.6%, 26.7%, 14.5%, and 15.5%, respectively.Source: §4.2 · p. 8

4.3 Memory Operations

  • Models abstain more successfully on Paralinguistic and Environmental questions (34.3% and 34.2%) than on Speech Semantics and Speaker Identity questions (19.9% and 16.2%).The authors interpret this reversed answerable-question pattern as difficulty processing acoustic information rather than genuine recognition that evidence is absent.Source: §4.3 · p. 8, Figure 5 · p. 8

4.4 Controlled History Length and Error Analysis

  • From 8K to 32K, mean accuracy falls from 40.1% to 33.0% for proprietary models and from 26.6% to 21.9% for open-weight models.Because questions and required evidence stay fixed across budgets, the decline is attributed to the growing surrounding history; at 64K, retention is 70.3% for Speech Semantics, 69.8% for Paralinguistic Cues, 66.5% for Speaker Identity, and 63.2% for Environmental Sound.Source: §4.4 · p. 9, Figure 6 · p. 9
  • IE errors are dominated by binding failures (46%), while MSR errors center on evidence localization (48%) and operation execution (32%), and TET errors on localization (55%).Across evidence types, speaker errors are most often binding failures (48%), paralinguistic errors localization failures (63%), and environmental errors split between localization (41%) and unsupported answers (39%).Source: §4.4 · p. 9, Figure 7 · p. 9

5 Conclusion

VoxMem evaluates spoken conversational memory across acoustic evidence and memory operations, and finds that models handle lexical content better than audio-native information. Its results distinguish context-length degradation from evidence-type difficulty and identify distinct failure profiles.

5 Conclusion

  • VoxMem organizes spoken conversational memory across acoustic evidence and memory operations, spanning 15 evaluation scenarios at four controlled context lengths.Across 15 LALMs, models remember what was said better than speaker identity, vocal delivery, or environmental sounds.Source: §5 · p. 10
  • As histories grow, access to previously available evidence consistently degrades, but weaker memory types are not necessarily more sensitive to context length.Models track evolving states reliably only when changes are stated in words.Source: §5 · p. 10
  • Speaker-identity errors most often misattribute information to the wrong person, while paralinguistic errors more often reflect failure to retain the acoustic cue.The conclusion argues that persistent spoken interaction requires preserving non-lexical acoustic information and its associations with speakers, events, and sessions over time.Source: §5 · p. 10

A Task Definitions and Taxonomy

The taxonomy separates what information must be recovered from how historical evidence must be used. It defines distinct answerable operations and evidence types, with tasks designed around their different retrieval and temporal requirements.

A Task Definitions and Taxonomy

  • The taxonomy distinguishes the evidence-type axis, describing what must be recovered, from the memory-operation axis, describing how it must be used.It specifies valid task forms for Information Extraction, Multi-Session Reasoning, and Temporal Evolution Tracking.Source: A Task Definitions and Taxonomy · p. 13

A.1 Memory Operations and Subtypes

  • Information Extraction uses one answer-bearing historical session; Multi-Session Reasoning combines evidence across sessions independent of temporal order; Temporal Evolution Tracking depends on ordered states.Answer Refusal pairs with an answerable item after the evidence needed for a unique answer is removed, neutralized, or made ambiguous; the taxonomy includes 30 answerable evidence-type–subtype routes.Source: §A.1 · p. 13, Table 3 · p. 14

A.2 Evidence Types and Scope

  • Each question receives one primary evidence type based on answer-critical evidence: Speech Semantics, Speaker Identity, Paralinguistic Cues, or Environmental Sound.Only Speech Semantics is transcript-sufficient; answerable questions in the other three types require information not recoverable from lexical content alone.Source: §A.2 · p. 13, Table 4 · p. 15

A.3 Information Extraction

  • Information Extraction requires aligning a retrieval anchor, the relevant historical session, and target information; its valid forms exclude Speech-Semantics IE.When both retrieval and requested information are lexical, an item can reduce to local transcription or semantic fact retrieval rather than testing cross-modal associations.Source: §A.3 · p. 13, Table 5 · p. 15

A.4 Multi-Session Reasoning

  • Multi-Session Reasoning requires evidence from multiple sessions, but permuting those evidence sessions must leave the answer unchanged.Its subtypes are counting, matching or resolution, and aggregation or comparison; the evidence type determines the retrieval and grouping signal.Source: §A.4 · pp. 13, 16, Table 6 · p. 16

A.5 Temporal Evolution Tracking

  • Temporal Evolution Tracking uses ordered historical states, so changing their order or temporal anchors may change the answer.Its subtypes ask for the latest state, a specified historical state, or an ordered trajectory; timestamps are default anchors, with explicit speech time expressions also possible for Speech Semantics.Source: §A.5 · p. 16, Table 7 · p. 17

B Benchmark Construction and Validation Details

The appendix details how benchmark specifications become spoken dialogues and describes the construction and validation procedures.

B Benchmark Construction and Validation Details

B.1 Question and Evidence Specification

VoxMem constructs questions from structured evidence specifications, then realizes them as controlled multi-session audio histories. Paired acoustic controls, answer-refusal variants, and nested context budgets preserve answer-critical evidence while varying the surrounding history.

B.1 Question and Evidence Specification

  • Each candidate specifies its memory operation, evidence type, target, relevant session selection, answer, supporting cues, and validity requirements before dialogue generation.Deterministic checks verify schema, route, cardinality, references, labels, and recomputed answers before candidates proceed.Source: §B.1 · p. 17

B.2 Spoken Dialogue Generation

  • Separate user and evidence-blind assistant generators prevent answer leakage, while lexical constraints keep speaker, paralinguistic, and environmental cues recoverable only from audio.User turns are aligned to structured evidence; assistant replies cannot repeat answer-bearing details or explicitly describe target acoustic attributes.Source: §B.2 · p. 18, Table 8 · p. 18

B.3 Acoustic Realization and Controls

  • Fixed voice references, synthesis controls, and matched clean-audio pairs vary speaker, vocal state, or environmental sound while preserving non-target dimensions.Environmental sounds are mixed at a fixed 10 dB SNR; paired controls preserve the canonical transcript and other non-target acoustic dimensions.Source: §B.3 · pp. 18–19, Table 9 · p. 19

B.4 Answer Refusal Construction

  • Answer-refusal items derive from answerable items by removing, neutralizing, or obscuring evidence needed for a unique answer, with the query and context budget preserved.Of 166 constructed variants, 130 passed full validation and entered the benchmark.Source: §B.4 · p. 19, Table 10 · p. 19

B.5 Haystack, Filler, and Session Reuse

  • Evidence sessions carry answer-critical facts, haystacks supply plausible same-key distractors, and screened filler sessions provide ordinary context; reuse is capped.Haystacks must not support the gold answer, and fillers are screened against benchmark facts and answers.Source: §B.5 · p. 20

B.6 Controlled History Assembly

  • Each item appears in nested 8K, 16K, 32K, and 64K histories that keep its question, answer, and evidence fixed while adding distractors and filler.Matched distractors increase with budget; evidence and haystacks use the same positional policy, and temporal ordering required by tracking questions is preserved.Source: §B.6 · p. 20

C Evaluation Protocol

The evaluation uses native audio inputs, a shared timestamped conversation format, and a model-independent reference scale for history length. A rubric-based judge scores answerable and insufficient-evidence responses, with cross-checks for scoring reliability and context-length comparisons.

C.1 Evaluated Models and Exact Versions

  • Fifteen systems receive user turns as audio through their native interfaces, without transcripts or a speech-recognition front end.Exact model versions and inference settings are recorded or released with the code.Source: §C.1 · p. 22, Table 12 · p. 22

C.2 Input Formatting

  • All models receive one conversation followed by one query in an identical, chronologically ordered format, with timestamps as the only session-boundary markers.The final session contains only the spoken query and its timestamp.Source: §C.2 · pp. 22–23

C.3 Reference Context-Length Measurement

  • History length uses a model-independent Whisper reference scale because audio and text tokens are not commensurable and model tokenizers differ.Audio is measured at 50 Whisper tokens per second; histories use whole sessions rather than truncation.Source: §C.3 · p. 23

C.4 Scoring and Judge Rubric

  • A judge scores answerable items for supported semantic equivalence and refusal items for evidence-based abstention; capability-based refusals do not count as correct.The judge sees the question, reference answer, answer type, and response, but not the conversation history.Source: §C.4 · pp. 23–24, Table 13 · p. 24

C.5 Judge Cross-Validation

  • The two judges agree on 97.95% of 8K predictions (κ = 0.953), with unchanged model ranking (ρ = 0.999) and 0.86 pp mean absolute per-model difference.The production judge favors Gemini outputs by 0.88 pp; the strongest three models are within 0.9 pp at 8K, so their ordering there is not meaningful.Source: §C.5 · p. 24

C.6 Models Included in the 64K Analysis

  • Ten systems completed a full 64K history, and they form the model set for reported 64K results.Where Figure 6 compares budgets, it uses the same ten models at every budget.Source: §C.6 · p. 24

D Additional Results and Robustness

  • Appendix results reuse the main text’s judged predictions; within-model budget columns use identical judged items, while Figure 8 uses a shared ten-model, four-budget subset.Figure 8 values therefore differ from Figure 5’s 32K average over fifteen models.Source: D Additional Results and Robustness · p. 24

D.1 Full 64K Results

At 64K, speech semantics remains the strongest evidence type, while audio-native evidence types score lower. Across history lengths, operation-by-evidence patterns persist even as accuracy declines.

D.1 Full 64K Results

  • At 64K, proprietary models score 51.7 on speech semantics against 26.0 for open-weight models, while the groups differ by under five points on each audio-native type.Speech semantics is highest, speaker identity second, and paralinguistic cues and environmental sound lowest and close together.Source: §D.1 · pp. 24, 26, Table 15 · p. 25

D.2 Operation × Evidence Across History Lengths

  • At 8K, 16K, and 64K, speech-semantics Temporal Evolution Tracking is among the strongest cells, while acoustic-state tracking is the weakest.Figure 8 reports means by operation and evidence cell for ten 64K models on items judged at all four budgets; Speech-Semantics IE is excluded by design.Source: §D.2 · p. 26, Figure 8 · p. 26
  • Longer histories lower accuracy across cells without changing the pattern; Answer Refusal is highest for paralinguistic and environmental evidence at every length.Among answerable operations, Multi-Session Reasoning varies least across the three audio-native evidence types.Source: §D.2 · p. 26

D.3 Answerable versus Answer Refusal Performance

The appendix separates answerable performance from refusal, measures paired context-length changes, and details how incorrect answers are attributed to stages in the memory process.

D.3 Answerable versus Answer Refusal Performance

  • Answerable and answer-refusal model rankings are nearly independent (ρ = −0.02), and the two top overall models are also strongest on answerable items.Gemini-2.5-Pro ranks second to last on answerable items but first on refusal items; always abstaining would score 16.3% overall and 0% on answerable items.Source: §D.3 · pp. 26–27

D.4 Paired Uncertainty for Context Scaling

  • Across every model and evidence type, the 8K-to-64K drop excludes zero; retention is highest for speech semantics (0.703) and paralinguistic cues (0.698).Speaker identity retains 0.665 and environmental sound 0.632; absolute drops differ partly because starting accuracies differ.Source: §D.4 · p. 27, Table 18 · p. 27

D.5 Baseline Difficulty versus Context Sensitivity

  • Retention has no detectable relation to 8K accuracy (ρ = 0.03, p = 0.84), unlike percentage-point change (ρ = −0.77, p < 0.001), whose association is largely due to the floor.The comparison distinguishes low baseline accuracy from sensitivity to added history across 40 model-by-evidence cells.Source: §D.5 · p. 28, Figure 9 · p. 28

E Error Analysis Details

  • The error analysis covers 4,740 attempts on 474 answerable audio-native items across ten 64K systems; 2,213 committed incorrect answers are attributed.Refusals, empty generations, and mid-sentence answers are excluded before attribution.Source: E Error Analysis Details · p. 28

E.1 Error Attribution Taxonomy

  • Errors are assigned to the earliest failed stage—recognition, binding to speaker or session, then operation—while unsupported answers apply only if no stage rule fits.The taxonomy’s five classes are derived from audit records using fixed rules, not from the response alone.Source: §E.1 · pp. 28–29, Table 19 · p. 29

E.2 Attribution Procedure

  • An auditor records factual properties of each response without seeing the error taxonomy; code then assigns a class using a fixed ordered rule set.For example, identifying the correct cue but failing the requested operation is classified as operation execution, not recognition.Source: §E.2 · p. 29

E.3 Breakdown by Question Subtype

At 64K, subtype-level errors concentrate in different ways: Cue-to-Fact in binding, Counting in operation execution, and Aggregation in recognition. Table 20 provides unweighted counts alongside Figure 10’s macro-averaged proportions.

E.3 Breakdown by Question Subtype

  • 78% of Cue-to-Fact errors are binding failures, versus 0–14% in every other subtype.Cue-to-Fact uses an acoustic cue to identify the session whose fact must be retrieved.Source: Figure 10 · p. 30, §E.3 · p. 31
  • 54% of Counting errors are operation errors, versus 13% in Matching, at most 12% elsewhere, and 1% (1 of 101) in Aggregation.Only four errors report a real tracked-attribute value at the wrong time; most temporal errors occur before the state sequence is assembled.Source: §E.3 · p. 31
  • 89% of Aggregation errors are recognition failures; responses compare thread topics rather than the cues.Source: §E.3 · p. 31

E.4 Absolute Counts

  • Table 20’s unweighted counts differ from Figure 10’s macro-averaged proportions when errors are unevenly spread across systems; Aggregation and Latest state have the fewest cells.The proportions for those subtypes should be read with that smaller cell base in mind.Source: §E.4 · p. 31, Table 20 · p. 31

E.5 Representative Failure Cases

The audited cases illustrate recognition, localization, binding, and operation-execution failures, as well as unsupported answers that invent evidence. Some responses retrieve relevant content but attach it to the wrong session or mishandle the evidence after locating it.

E.5 Representative Failure Cases

  • A response to an Aggregation item described the two threads’ topics but named neither vocal delivery, leaving the acoustic cue unused.Source: §E.5 · pp. 31–32
  • For a Counting question, Audio-Flamingo-Next counted ten topic-matching chats instead of the two sessions meeting the speaker condition.Source: §E.5 · p. 31
  • In a Cue-to-Fact case, Gemini-2.5-Pro gave content from Session 26 even though the delivery cue identified a different session.Source: §E.5 · pp. 31–32
  • Operation errors included an incorrect count despite located members and a wrong comparison despite correctly described deliveries; another answer invented sound categories and counts.The examples span Counting, Matching, and Environmental Aggregation.Source: §E.5 · pp. 31–32

F Prompts

This appendix presents the prompts that shape benchmark content and scoring, including quality-control checks for semantic, acoustic, and full-history validity and standardized model instructions, answer and abstention scoring, and error attribution. The prompts distinguish text-only judgments from checks requiring audio or assembled histories, and analyze the source of already-incorrect responses without rejudging correctness.

F.1 Question and Evidence Generation Prompt

  • Answerable operations share a system prompt but use operation-specific user prompts; answer-refusal items are derived later without a generation prompt.The user prompt carries the generation request and four component schemas as runtime placeholders.Source: §F.1 · p. 32
  • Question generation requires a uniquely determined answer and individually necessary evidence, while preventing transcript-only shortcuts for non-semantic cues.Evidence necessity is operation-specific, including special rules for temporal tracking’s latest-state queries.Source: §F.1 · pp. 32–33
  • The question-generation prompt binds candidates to the supplied blueprint and schemas, fixing operation extensions, answer constraints, payloads, and evidence identifiers.It also prohibits demographic inference and alternative answer routes through redundant facts or wording.Source: §F.1 · p. 33

F.2 Dialogue Generation Prompts

  • The assistant-reply generator is evidence-blind and uses only the visible conversation prefix to produce one reply consistent with its reply contract.It avoids unnecessary echoes of personal details and comments on voice, identity, emotion, or environmental sound.Source: §F.2 · pp. 34, 36, 37
  • User-turn generation realizes each session’s assigned facts once in natural dialogue, without leaking acoustic targets or adding answer-changing details.It requires exact fact spans and aligns semantic evidence to text spans and acoustic evidence to audio turns or spans.Source: §F.2 · pp. 35–36

F.3 Quality-Control Prompts

  • The quality-control gates test answer validity from transcripts, acoustic necessity against paired controls, and full-history issues across nested budgets.The full-history gate uses every user-turn waveform in the 64K history and reports the earliest budget where a problem appears.Source: §F.3 · p. 37
  • The semantic gate checks unique answerability, designated-evidence necessity, semantic shortcuts, and paired refusal validity; audio-only AR changes are not applicable to text-only review unless text itself fixes the answer.For partial-evidence removal, deleted sessions are absent and the check applies in full.Source: §F.3 · p. 38
  • The acoustic gate requires primary audio to support the answer, the control to isolate the target dimension, and the ablation to break answerability; transcripts alone must not determine nonsemantic answers.Controls compare speaker binding, vocal cues, or environmental audio; semantic clue removal is used only when explicitly requested.Source: §F.3 · p. 39
  • The full-history gate checks for leakage, joint shortcuts, assistant echoes, answer-relevant contradictions, and unintended answer routes after all sessions are assembled.It reports the earliest affected context tier when a problem appears only after additional sessions are introduced.Source: §F.3 · pp. 39–40
  • Full-history refusal checks account for whether evidence was deleted, acoustically altered, or voice-swapped, and fail only if the answer remains inferable from surviving audio, sessions, or text.For replacement methods, judges compare the delivered AR audio with the pre-replacement rendition and check for acoustic-role collisions in other sessions.Source: §F.3 · p. 40

F.4 Evaluation and Scoring Prompts

  • Evaluation uses the abstention-permitted candidate instruction, while acoustic-evidence validation uses the no-abstention variant; both direct models to use the complete timestamped audio-and-text history.The permitted variant requires an evidence-based abstention when the conversation cannot determine one unique answer; the other requires a best answer.Source: §F.4 · pp. 40–41
  • The answerable-item judge scores semantic equivalence rather than exact wording, while requiring all elements and correct order for sets and ordered options.It rejects omissions of answer-critical content, incompatible answers, and abstentions; option responses must commit to exactly one matching option.Source: §F.4 · pp. 41–42
  • The AR judge accepts evidence-based abstentions that attribute uncertainty to insufficient conversational evidence, but rejects guesses, candidate lists, and safety- or capability-based refusals.A generic “I don’t know” is insufficient unless it identifies missing or insufficient information in the conversation.Source: §F.4 · p. 42

F.5 Error-Attribution Prompt

  • The error-attribution auditor examines wrong answers without seeing the error taxonomy, reports where the committed answer came from, and uses chronological evidence, distractor, and filler session labels.The prompt also encodes how turns sounded in transcripts because readers cannot hear the audio.Source: §F.5 · p. 42
  • The prompt requires a single JSON object recording the commitment, answer, cue recovery, source session, relation to the reference, and explanation.It explicitly instructs the evaluator to determine only where the response’s answer came from.Source: §F.5 · p. 43
  • Commitment labels distinguish specific answers from generic, off-type, absent, and truncated responses.“None” includes acknowledgements and flat denials despite a cue; “truncated” covers text that ends mid-answer or never reaches a final answer.Source: §F.5 · p. 43
  • Cue recovery is assessed independently of commitment, so a response may identify the relevant sound, delivery, or speaker but still fail the requested answer.For example, recognizing a sound does not mean the response supplied the count asked for.Source: §F.5 · p. 43
  • Source labels distinguish answers found in evidence sessions from answers drawn from distractor or filler sessions, absent answers, and no commitment.An invented inventory of sounds or speakers is classified as absent.Source: §F.5 · p. 43
  • Relations to the reference distinguish real earlier or later states from count errors, while other labels capture values taken from another session or unrelated answers.Earlier- or later-state labels apply only when the answer is a real value of the same thing at a different time; count-off applies when the counted members are real.Source: §F.5 · p. 43
  • The route example labels “canal route” as a specific answer from another session, not the evidence-session reference “market route.”The distractor session supplies the canal route, while the evidence session with water drops supplies the market route.Source: §F.5 · p. 43
Loading 2609.32607v1…