PaperlayerView exact revision on arXiv

Source-linked AI summary

HEAR Who Said What: Unlocking Speaker-Attributed Reasoning via Counterfactual Voice Grounding

Dongwook Lee, Sangkwon Park, Eunwoo Song, Che Hyun Lee, Youngho Cho, Junho Kim, June Young Yi, Heeseung Kim, Sungroh Yoon

arXiv:2608.29120v1cs.CLcs.AIcs.SD

TL;DR

The paper addresses limited evidence about whether speech language models can attribute utterances to speakers and reason over speaker identities. It introduces HEAR to diagnose foundational capabilities and trains A2R on CASH to prioritize vocal cues. A2R substantially improves HEAR performance and transfers to unseen multi-speaker tasks, while explicit transcript generation introduces latency.

  • Problem

    Speech language models increasingly operate in multi-speaker environments, but their ability to attribute speech correctly and reason over speaker identities remains unclear.

  • Method

    The paper introduces HEAR and trains the 30B A2R model with CASH voice-cloned hard negatives and a transcription-first objective.

  • Results

    A2R substantially improves performance on HEAR and transfers to unseen multi-speaker tasks requiring speaker attribution.

  • Takeaways & Limitations

    The findings establish speaker attribution as a key step toward multi-party auditory comprehension.

  • Takeaways & Limitations

    Explicit transcript generation maximizes attribution performance but introduces latency that limits real-time applications.

Abstract

from arXiv · show

Speech Language Models (SLMs) are increasingly deployed in multi-speaker environments, yet their ability to attribute speech to the correct speaker and reason over speaker identities remains unclear. Hence, we introduce HEAR, a conceptually hierarchical benchmark diagnosing the foundational capabilities of speaker-attributed reasoning, comprising 2.4K human-verified samples from 887 diverse multi-party audio clips. Evaluating 20 leading SLMs on HEAR reveals they struggle with these foundational tasks, often relying on semantic priors rather than actual vocal cues. To address this, we present A2R, a 30B model optimized on Counterfactual Audio with Speaker-level Hard negatives (CASH), a dataset designed to guide the model to prioritize acoustic vocal cues over linguistic signals. A2R achieves strong performance on HEAR and exhibits zero-shot generalization to diverse multi-speaker downstream tasks, demonstrating that learned speaker attribution unlocks the model's latent capacity for speaker-aware reasoning. All resources are available at https://attributetoreason.github.io/AttributeToReason/

1 Introduction

HEAR diagnoses foundational failures in speaker-attributed reasoning, while CASH and A2R train models to prioritize vocal cues before reasoning over speaker identities.

  • Motivation: Prior evaluations often tested speaker attribution only through downstream tasks, making foundational failure points difficult to localize.The paper distinguishes failures in voice discrimination, utterance attribution, and downstream reasoning.
  • HEAR: HEAR evaluates discrimination, attribution, and reasoning using 887 real-world audio clips and 2.4K human-verified multiple-choice question-answer pairs.Its taxonomy follows Erber’s auditory hierarchy from perceptual discrimination to higher-level comprehension.
  • Findings and intervention: Most leading SLMs fail basic attribution and over-rely on semantic priors when semantic context is available.CASH addresses this by decoupling acoustic identity from preserved linguistic content through speaker-swapped hard negatives.
  • A2R: A2R strengthens speaker attribution with GRPO and a transcription-first objective that produces speaker-tagged transcripts before reasoning.The approach aims to preserve the base model’s inherent reasoning ability while improving vocal grounding.
  • Results: A2R excels on HEAR and transfers robustly to downstream multi-speaker benchmarks requiring speaker attribution.The results support speaker attribution as a key mechanism for speaker-aware reasoning in multi-speaker environments.

2 HEAR Benchmark

HEAR organizes speaker-attributed reasoning hierarchically, testing voice discrimination, content-to-voice binding, and reasoning under counterfactual voice changes.

  • Benchmark structure: HEAR mirrors human auditory processing through three dimensions: Discrimination, Attribution, and Reasoning.The benchmark uses multiple-choice question answering and separately analyzes whether overlapping speech is needed.
  • Discrimination: Discrimination tests voice identity clustering and temporal boundaries through voice cardinality, voice localization, and voice change detection.These tasks measure speaker count, target-voice activity intervals, and speaker transition points, including simultaneous speech.
  • Attribution: Attribution binds semantic content to voice identity in both directions: text to audio and audio to text.CVA selects a speaker for a transcribed utterance, while VCA selects text spoken by a reference voice from five candidates.
  • Reasoning: Reasoning pairs each original clip with a transcript-preserving voice-swapped variant whose correct answer flips, exposing voice-blind models.The reasoning axis includes identity, temporal, and quantitative reasoning over speaker-attributed utterances.
  • Dataset: HEAR contains approximately 2.4K queries from 887 clips totaling roughly 23 hours, with an average of 4.05 speakers per clip.Queries undergo rule-based generation followed by manual verification, providing two stages of human validation.

3 Training

The training approach counters semantic reliance with voice-swapped hard negatives and trains A2R to transcribe speaker-tagged audio before reasoning.

  • Motivation: Semantic hallucination occurs when SLMs favor textual priors over acoustic cues in multi-party audio.Because any participant may make a statement, semantic plausibility alone cannot determine who actually spoke.
  • CASH: CASH is a 60K-scale multi-party dataset designed to decouple acoustic identity from semantic content.It spans all HEAR task dimensions and uses voice-converted versions of the same transcript while preserving chronology and other speech content.
  • A2R: A2R is optimized via GRPO to enhance speaker attribution while preserving the baseline model’s general performance.The model first transcribes input audio with speaker tags, then generates reasoning and a final answer from that transcript.
  • Reward design: Training rewards speaker-tagged transcription, final-answer correctness, and response format.The transcription reward decomposes into attribution accuracy, speaker count, and speaker-order preservation.
  • Pipeline: Figure 2 connects voice-swapped data construction, speaker-attribution tasks, and transcription-first reasoning.Its three panels show CASH generation, an attribution-dependent query, and the model’s transcript-then-reason pipeline.

4 Experiments

The experiments evaluate leading SLMs on foundational and downstream speaker-attributed reasoning tasks, including tests designed to separate acoustic grounding from semantic shortcuts. They also assess transfer beyond HEAR and preservation of dyadic conversational ability.

  • Experimental Setup: The evaluation covers 20 leading SLMs across omni-modality, speech/audio language, and proprietary model categories.Qwen3-Omni-30B-A3B-Instruct is selected as the base model for developing speaker-attributed reasoning capabilities.
  • Experimental Setup: HEAR experiments include semantic-hallucination pairs that preserve transcripts while requiring different answers after speaker swaps.Pairs enforce transcript fidelity, speaker similarity above 0.7, answer flips, and human verification of the target speaker.
  • Evaluation Criteria: The paired accuracy metric counts Reasoning predictions only when both an original clip and its semantic-hallucination counterpart are answered correctly.This metric is used to test whether reasoning remains correct across voice-swapped counterparts.
  • Downstream Benchmarks: Zero-shot transfer is evaluated on WDYL and Gaokao, where query voices must be associated with the correct speaker or earlier dialogue character.WDYL uses Identity-Critical Questions, while Gaokao synthesizes the final query in a randomly selected preceding character’s voice.
  • Downstream Benchmarks: Find the Spy requires tracking which player produced each clue before identifying the semantically inconsistent spy description.Players describe assigned words in random speaking order, making speaker attribution necessary for the downstream vote.
  • Downstream Benchmarks: VoiceBench provides an additional evaluation of whether speaker-attributed modeling preserves dyadic spoken-language ability.The benchmark covers eight benchmarks in dyadic conversational settings.

5 Results & Analysis

HEAR exposes weaknesses in speaker attribution and voice-grounded reasoning, while A2R transfers gains across unseen and human-voice tasks with limited general-capability trade-offs.

  • Leading SLM limitations: Most leading SLMs perform near chance on discrimination and attribution, with overlapping speech further degrading performance across six baselines.A2R largely mitigates this limitation, reaching performance substantially similar to Gemini Flash.
  • Semantic hallucination: Strong reasoning scores can overstate voice-grounded ability because paired accuracy falls sharply when models must answer both original and semantic-hallucination clips.The pattern also appears in proprietary Gemini models, indicating reliance on semantic cues.
  • Unseen-task transfer: +36.4 (WDYL), +22.6 (GAOKAO), and +19.0 (FTS) percentage points are A2R’s gains on unseen attribution-critical benchmarks.These results support transfer without task-specific training.
  • Human-voice transfer: +48.0, +20.0, and +30.0 percentage points are A2R’s improvements over the base model on WDYL, GAOKAO, and FTS using human recordings.The evaluation compares 100 synthetic samples with corresponding human recordings.
  • Ablations: Removing CASH hard negatives reduces HEAR reasoning pair accuracy from 60.0 to 43.0 for one seed, while removing structured reasoning constraints also degrades performance and pair consistency.Replacing the objective with LoRA-based SFT produces weaker HEAR and VoiceBench results.
  • General capability and prompting: A2R remains broadly on par with the baseline across eight general dyadic benchmarks, indicating only a marginal impact on general capability.Transcript-first prompting helps further, but A2R still outperforms baselines on most benchmarks without that scaffold.

6 Related Works

Prior multi-speaker systems and benchmarks address parts of speaker attribution but leave speaker-content reasoning insufficiently isolated and explored.

  • Speaker attribution: Speaker diarization identifies who spoke when, while multi-talker ASR identifies what was said; newer recognizers add speaker information or instruction-guided transcription.These approaches mainly produce speaker-labeled transcripts rather than reasoning over speaker-content links.
  • Speaker-attributed reasoning: Speaker-attributed reasoning links what was said to who said it, supporting questions about utterance ownership, speaking order, or comparative statements.The capability extends beyond producing speaker-labeled transcripts.
  • Benchmark gaps: Broad benchmarks include multi-speaker scenarios, but speaker attribution is only partially isolated and often appears as a small subset among many evaluation axes.This motivates a benchmark focused on foundational speaker-attributed understanding.

7 Conclusion

The paper introduces HEAR to diagnose speaker-attributed reasoning failures, then uses CASH and A2R to improve vocal grounding and transfer to unseen multi-speaker tasks.

  • HEAR evaluates foundational speaker-attributed reasoning and reveals that leading SLMs often rely on semantic priors instead of vocal evidence.
  • CASH uses voice-cloned hard negatives to decouple acoustic cues from semantic content, guiding models toward vocal evidence.The dataset contains 60K-scale voice-swapped examples.
  • A2R explicitly learns speaker attribution and substantially improves performance on HEAR and unseen multi-speaker tasks.
  • The findings position speaker attribution as a key step toward multi-party auditory comprehension.

Limitations

Transcript-first reasoning maximizes speaker attribution performance but adds latency that limits real-time applications. A2R still outperforms baselines without externalized transcripts, though performance is slightly lower.

  • Transcript-first reasoning maximizes speaker attribution performance but introduces latency that limits real-time applications.
  • A2R significantly outperforms baselines without transcript generation, albeit with slightly lower performance.
  • Future work should bridge the latency–performance gap through implicit reasoning over internal representations.

Ethics Statement

The benchmark combines controlled synthetic voice manipulation with human-verified source data, while access and use are restricted because voice cloning creates consent, privacy, and dual-use risks.

  • The synthetic speech component is treated as a controlled research resource because voice cloning may enable impersonation, fraud, harassment, and speaker re-identification.
  • Source licenses do not necessarily constitute consent to generate novel utterances in individuals’ voices, and no additional task-specific consent was obtained.
  • Synthetic waveforms are gated to approved non-commercial research, with identity, affiliation, intended use, and a Data Use Agreement required for access.
  • Synthetic utterances are documented as artificial samples and are not statements, beliefs, endorsements, or views of source speakers.
  • CASH constructs counterfactual multi-party clips by replacing target utterances with voice-converted versions while preserving transcripts and chronology.
  • HEAR questions undergo automated generation followed by manual review, with 887 clips and 2,395 questions surviving verification.
  • The benchmark includes discrimination and attribution tasks involving voice counts, localization, transitions, overlaps, and transcript–voice matching.

B.5 Benchmark Statistics

HEAR contains 887 clips spanning three domains and 2,395 questions across discrimination, attribution, and reasoning, with deliberately varied source and genre coverage.

  • 887 clips comprise approximately 22.8 hours of audio and 290 unique speakers across the three source domains.
  • VoxMM contributes 40.6% of clips, ICSI 32.2%, and AMI 27.2%, so no single domain dominates the benchmark.
  • 2,395 questions comprise 627 Discrimination, 608 Attribution, and 1,160 Reasoning items.
  • The eight subdimensions include VC, VL, VCD, CVA, VCA, QR, IR, and TR, with QR the largest at 23.4%.
  • VoxMM’s genre mix is headed by Remote and Documentary, while several other genres form a long tail.
  • Across domains, the benchmark tracks clip count, source recordings, speakers, and average duration, with clip durations evenly distributed.

C Details of Benchmarks Where Speaker Attribution Is Essential

The paper evaluates speaker attribution through benchmarks where answering correctly requires identifying and tracking who is speaking, not merely understanding content. These benchmarks span first-person reference resolution, speaker-conditioned listening comprehension, and multi-speaker voice tracking.

  • WDYL (What Do You Like?): WDYL tests whether models resolve first-person references using the question speaker’s identity.The question is spoken in one participant’s cloned voice, so the referent of “my” depends on identifying that speaker.
  • WDYL (What Do You Like?): 793 questions over 793 clips comprise WDYL, with 50% used as the effective random-choice baseline.Each question has three choices, but the transcript eliminates one distractor, leaving two effective alternatives.
  • GAOKAO: GAOKAO uses natural English listening-comprehension dialogues followed by a first-person question in one speaker’s cloned voice.The benchmark contains 93 questions over 93 clips and uses two answer choices with a 50% random-guess accuracy.
  • FTS (Find the Spy): FTS requires tracking speakers across introductions and shuffled, unnamed clues to identify the player assigned as the spy.Because clues are presented in a different order and without names, textual content alone does not identify their producers.
  • FTS (Find the Spy): 400 FTS questions cover three- to six-player games, with chance accuracies from 33.3% to 16.7% as player count increases.The benchmark contains 3.67 hours of audio, with 100 games for each player count.

D Experiment Details

The experiments evaluate 20 SLMs across omni-modality, spoken/audio, and proprietary model families, using standardized inference settings and speaker-focused benchmarks. They also specify the speaker-order reward, fine-tuning configuration, and VoiceBench evaluation dimensions.

  • Model Evaluation: 20 leading SLMs are evaluated across omni-modality, spoken/audio, and proprietary model families.The listed systems include MiniCPM-o, Qwen Omni, spoken-language models, Gemini-3, and GPT-4o-audio-preview.
  • Experimental Settings: Open-source baselines use repository-specified hyperparameters, while proprietary systems use default API settings.Open-weight models are served through vLLM or Hugging Face inference paths, and closed-source endpoints are queried with default settings.
  • Speaker-Order Reward: Rord is a length-normalized longest-common-subsequence score for comparing predicted and ground-truth speaker-label sequences.The dynamic program matches correctly ordered labels and skips hallucinated or missing turns during alignment.
  • Speaker-Order Reward: Misattribution, insertion, and deletion examples each receive Rord = 0.8, approximately 0.83, and 0.8, respectively, under the paper’s five-turn example.An extra predicted turn increases the denominator, while a misattributed or deleted turn reduces the matched subsequence.
  • Fine-Tuning: Fine-tuning applies rank-64 adapters to language-model projections while freezing audio/vision encoders and modality aligners.Training runs for one epoch with effective batch size 64 and bfloat16 precision; expert parallelism is set to EP=4.
  • Downstream Evaluation: VoiceBench evaluates A2R on open-ended QA, reference-based QA, multiple-choice QA, instruction following, and safety.The benchmark combines synthetic spoken instructions with real human-recorded speech and assesses conversational, reasoning, compliance, and refusal behavior.
  • Paired Reasoning Accuracy: Reasoning uses paired accuracy, counting an evaluation unit as correct only when both original and semantic-hallucination clips are answered correctly.The overall average weights discrimination, attribution, and reasoning by their respective numbers of evaluation units, counting each reasoning pair once.
Loading 2608.29120v1…