PaperlayerView exact revision on arXiv

Source-linked AI summary

Voices as Handles: Reasoning about Speaker Identity with Frozen Text LLMs

Runqiu Xu, Zhisheng Zheng, David Harwath

arXiv:2609.38501v1eess.AScs.SD

TL;DR

Multi-user voice agents need to link facts to speakers across sessions, but transcripts alone do not provide acoustic identity and existing benchmarks may retain cues that obscure whether bindings are used. The paper introduces projected Speaker Handles for a frozen text LLM and evaluates them on SpeakerBind, where they achieve 70.40% compared with 71.88% using ground-truth speaker labels.

  • Problem

    Transcripts alone do not expose acoustic speaker identity, and existing benchmarks may retain textual cues that make it difficult to establish whether models use speaker–content bindings.

    Source: §1 · p. 1

  • Method

    A lightweight projector maps utterance-level speaker embeddings into reusable soft-token Speaker Handles, trained with a three-stage curriculum for use alongside transcripts by a frozen text LLM.

    Source: §1 · p. 1, §2.1 · p. 2

  • Results

    70.40% macro-averaged accuracy on SpeakerBind versus 71.88% using ground-truth speaker labels; on Vox1-O/E, Speaker Handles achieve 98.36% accuracy and on Vox1-H, 97.40%.

    Source: §4.2 · p. 4

  • Takeaways & Limitations

    Speaker Handles provide reusable acoustic speaker references for cross-session reasoning, with performance close to textual-label results on SpeakerBind.

    Source: §1 · p. 1, §4.2 · p. 4

    Existing benchmark performance alone does not establish use of speaker–content bindings because unlabeled transcripts can retain cues revealing speaker identity or answer-relevant structure.

    Source: §1 · p. 1

Abstract

from arXiv · show

Multi-user voice agents must track who said what across dialogue sessions. Text LLMs are attractive backbones for such agents, but transcripts alone do not expose acoustic speaker identity, leaving the model without a persistent reference for linking information to speakers across sessions. We address this gap by introducing Speaker Handles, soft-token representations that expose acoustic speaker identity to a frozen text LLM for cross-session speaker-dependent reasoning. A three-stage curriculum trains a lightweight projector, with fewer than 0.1% of the backbone's parameters, to map speaker embeddings into these handles. Establishing whether the resulting handles truly support cross-session speaker-dependent reasoning is challenging with existing benchmarks because textual cues can partially reveal fact ownership. We therefore present SpeakerBind, a controlled shared-agent benchmark in which overlapping facts across users require correct cross-session speaker attribution. Speaker Handles achieve 97.40-98.36% accuracy on VoxCeleb1 and 70.40% on SpeakerBind, close to the 71.88% topline. These results show that the proposed Speaker Handles provide an efficient way to integrate acoustic speaker identity into frozen text LLMs for speaker-content reasoning.

1. INTRODUCTION

The introduction motivates reusable acoustic speaker references for frozen text LLMs, presents Speaker Handles and SpeakerBind, and reports performance close to textual speaker labels.

1. INTRODUCTION

  • Multi-user voice agents must link utterances to user identities and maintain those associations across turns and sessions.Prior continuous prompts and lightweight connectors do not require speaker representations to act as reusable references across utterances.Source: §1 · p. 1
  • Speaker Handles are soft-token representations projected from utterance-level speaker embeddings and supplied alongside transcripts instead of textual speaker labels.The speaker encoder and text LLM remain frozen while a lightweight projector is trained with a three-stage curriculum.Source: §1 · p. 1
  • Existing benchmarks may retain transcript cues that reveal speaker identity or answer-relevant structure even after textual speaker labels are removed.Strong performance alone therefore does not establish that a model uses supplied speaker–content bindings.Source: §1 · p. 1
  • SpeakerBind requires binding facts to the appropriate speakers across sessions, addressing the risk that transcript cues can make existing benchmarks inconclusive.Its controlled interventions corrupt speaker–content bindings to test whether handles act as persistent references.Source: §1 · p. 1
  • Speaker Handles approach ground-truth textual-label performance, while incorrect speaker assignments cause performance to collapse.The paper presents a three-stage curriculum and SpeakerBind as complementary contributions for evaluating speaker-dependent reasoning.Source: §1 · p. 1

2. LATENT SPEAKER HANDLES FOR FROZEN LLMS

The method projects utterance-level speaker embeddings into soft-token handles used with transcripts by a frozen text LLM. A three-stage curriculum trains the projector to support speaker identity operations and content binding.

2.1. Speaker Projector

  • A frozen speaker encoder extracts each utterance embedding, and a trainable projector maps it to K language-model-space vectors called a Speaker Handle.The handle replaces a textual speaker label and can also serve as a query-speaker reference, roster entry, or prompt candidate.Source: §2.1 · p. 2
  • At inference, the projected handle is interleaved with its transcript so the frozen LLM can match recurring speakers and reason over their earlier content.The speaker encoder and language model remain frozen; only the projector is optimized.Source: §2.1 · p. 2, §2.2 · p. 2

2.2. Learning Speaker Handles

  • The three-stage curriculum trains on synthetic dialogue-like sequences, progressing from conditioned retrieval and verification to prompt variation, then broader identity–content operations.Stage 1 uses query- and anchor-conditioned tasks; Stage 2 varies handle presentation, and Stage 3 adds global, temporal, set, and error-detection operations while replaying earlier tasks. Table 1 summarizes training inputs and representative tasks.Source: §2.2 · p. 2, Table 1 · p. 2

3. SPEAKERBIND: CROSS-SESSION SPEAKER-CONTENT BINDING BENCHMARK

The section motivates SpeakerBind by showing that unlabeled transcripts can retain speaker-configuration cues, then describes a controlled shared-agent benchmark built around overlapping facts and cross-session attribution.

3.1. Residual Speaker Cues in Existing Benchmarks

  • Transcript-only participant counts were exact for approximately 34% of sampled dialogues and within one participant for 72%, suggesting residual speaker-configuration information.The study sampled 20 dialogues from each of six datasets, for 120 dialogues total; participant counting does not directly test speaker attribution.Source: §3.1 · p. 2

3.2. Benchmark Construction

  • SpeakerBind interleaves independently generated sessions from multiple users who share topics and predicates but report different facts, requiring attribution to the query speaker.Questions require recovering that speaker’s information while excluding competing facts from other users.Source: §3.2 · p. 3
  • The benchmark tests aggregation of a user’s facts, tracking ordered state updates across sessions, and linking a specified anchor to that speaker’s later follow-up.The task designs include distractors that share topics, values, or related actions, so topical overlap alone does not identify the target history.Source: §3.2 · p. 3
  • Task-specific fact templates define topics, predicates, and candidate values; independently generated dialogues are assigned to users and assembled into shared histories.Reference answers are derived from those assignments and the underlying facts and relations.Source: §3.2 · p. 3

4. EXPERIMENTS

The experiments evaluate whether projected Speaker Handles preserve speaker identity and support downstream reasoning, including controlled cross-session speaker attribution. Handles approach textual-label performance on SpeakerBind and existing benchmarks, while intervention and ablation results test whether performance depends on recurring speaker identity and staged training.

4.1. Setup

  • Experiments use a frozen Qwen3.5-9B and VoxBlink2 speaker encoder with a trained projector, comparing handles against textual-label, transcript-only, and cascaded-identification conditions.Existing benchmark evaluations cover VoxCeleb1, M3-SLU, FANToM, and NSF-QA; FANToM participants are mapped to held-out speaker identities.Source: §4.1 · p. 3
  • SpeakerBind comprises 149 task templates and 1,540 session specifications, with 1,000 histories per task and 6,000 questions total.Its Aggregation, Follow-up Linking, and Temporal tasks span multi-user histories averaging 138–187 turns.Source: §4.1 · pp. 3–4

4.2. Identity Preservation and Downstream Reasoning

  • 70.40% macro-averaged accuracy on SpeakerBind, versus 71.88% with ground-truth speaker labels, puts Speaker Handles 1.48 points below the topline.On Follow-up Linking, the gap is 0.10 points; templates and cross-session compositions were excluded from projector training.Source: Table 2 · p. 4, §4.2 · p. 4
  • 98.36% accuracy on Vox1-O/E and 97.40% on Vox1-H accompany consistent near-topline results across existing downstream benchmarks, with a slight M3-SLU topline exceedance.The reported benchmarks also include FANToM Fact QA and NSF-QA summarization.Source: §4.2 · p. 4
  • Reassigning answer-relevant turns sharply degrades accuracy across all three SpeakerBind tasks; Same approaches task-specific chance, while Distinct substantially degrades performance.Reassignment changes answer-relevant speakers while keeping the query speaker fixed; Same removes between-speaker distinctions, and Distinct removes cross-turn identity recurrence.Source: §4.2 · p. 4, Table 2 · p. 4

4.3. Ablation Study

  • The staged curriculum improves speaker verification and Aggregation after Stage 2, then further improves Temporal reasoning without sacrificing overall downstream performance.Training all tasks at once sharply degrades downstream performance despite substantial speaker-verification accuracy.Source: §4.3 · p. 4, Table 3 · p. 4

5. CONCLUSION

The conclusion presents Speaker Handles as a lightweight projector-based interface for using utterance-level speaker embeddings with a frozen text LLM. The handles approach textual-label performance on existing benchmarks and SpeakerBind, with staged training and binding interventions supporting their use beyond speaker discrimination alone.

5. CONCLUSION

  • A lightweight projector maps utterance-level speaker embeddings into latent Speaker Handles usable by a frozen text LLM.A staged curriculum supports identity operations and content binding; SpeakerBind evaluates cross-session attribution across Aggregation, Temporal, and Follow-up Linking.Source: §5 · p. 4
  • Speaker Handles approach textual-label performance on existing benchmarks and SpeakerBind, while ablations identify staged training as critical for downstream usability beyond speaker discrimination.Reassigned bindings drive accuracy toward the counterfactual expectation; same-speaker and distinct-speaker-per-turn controls also degrade performance.Source: §5 · p. 4

6. COMPLIANCE WITH ETHICAL STANDARDS

The study uses existing speech and dialogue datasets and synthetic data, without recruiting participants or collecting new human-subject data.

6. COMPLIANCE WITH ETHICAL STANDARDS

  • The study uses pre-existing speech and dialogue datasets and synthetic data, with no recruited participants or newly collected human-subject data.Source: §6 · p. 5

7. CONFLICT OF INTEREST

The authors report no relevant financial or nonfinancial interests.

7. CONFLICT OF INTEREST

  • The authors disclose no relevant financial or nonfinancial interests.Source: §7 · p. 5
Loading 2609.38501v1…