PaperlayerView exact revision on arXiv

Source-linked AI summary

AudioLens: Multi-Perspective Speech Clustering with Reasoning Audio-Language Models

Wenjun Huang, Qiaosong Chu, Tiger Shao, Pengfei Zhang, Yutong Song, Hanning Chen, Yezi Liu, Weiyi Wu, SungHeon Jeong, Ryozo Masukawa, Sanggeon Yun, Yang Ni, Jiang Gui, Mohsen Imani

arXiv:2608.25177v1cs.SDcs.AI

TL;DR

Existing audio-clustering approaches are limited by fixed acoustic metrics or ASR-based text representations when users need linguistic and paralinguistic perspectives. The paper introduces perspective-conditioned clustering, AudioLens-Bench, and AudioLens-R1, an end-to-end model trained with reasoning distillation and preference optimization. AudioLens-R1 consistently outperforms baselines, improving overall ARI by 12.99 points and V-measure by 11.62 points.

  • Problem

    Existing methods cannot flexibly reorganize the same speech collection under user-specified perspectives that may depend on both linguistic and paralinguistic cues.

  • Method

    The paper introduces perspective-conditioned audio clustering, AudioLens-Bench, and AudioLens-R1 trained with reasoning distillation and preference optimization.

  • Results

    AudioLens-R1 consistently outperforms all baselines, improving overall ARI by 12.99 points and V-measure by 11.62 points.

  • Takeaways & Limitations

    Native audio-language models can support flexible, perspective-conditioned structure discovery over speech collections.

  • Takeaways & Limitations

    The number of clusters is not provided as input and must be inferred by the model.

Abstract

from arXiv · show

Audio clustering is a fundamental task for organizing rapidly growing speech collections, supporting applications such as conversational analysis and speech-driven discovery. However, existing methods rely on fixed acoustic similarity metrics or ASR-based text pipelines, limiting their ability to reorganize the same audio collection under different user-specified perspectives, especially when clustering depends on both linguistic and paralinguistic cues. We introduce audio multi-perspective clustering, where a model directly partitions speech recordings according to a natural-language perspective while inferring both the number of clusters and their assignments. To study this setting, we construct AudioLens-Bench, a benchmark spanning multiple application domains and evaluating both in-perspective and cross-perspective generalization. We further propose AudioLens-R1, an end-to-end large audio-language model trained with reasoning distillation and preference optimization. Experiments show that AudioLens-R1 consistently outperforms all baselines, improving overall ARI by 12.99 points and V-measure by 11.62 points. These results demonstrate the promise of native audio-language models for flexible, perspective-conditioned structure discovery over speech collections.

1 Introduction

AudioLens frames clustering as perspective-conditioned structure discovery over speech, addressing the limits of fixed acoustic metrics and transcript-only pipelines. It introduces a benchmark and an end-to-end model for clustering the same collection under different linguistic and paralinguistic perspectives.

  • Motivation: Users may need to group recordings by communicative intent, affective state, speaker traits, or background context.These criteria require reasoning over linguistic content and paralinguistic cues.
  • Motivation: Existing methods either use fixed similarity metrics or text-only representations that discard how speech is delivered.ASR captures what is said but loses cues such as prosody and speaker traits.
  • Task formulation: Audio multi-perspective clustering uses a natural-language perspective to determine both the number of clusters and each recording’s assignment.The same audio collection can yield different valid partitions under different perspectives.
  • Benchmark: AudioLens-Bench evaluates diverse domains and both familiar and unseen clustering perspectives.Its held-in and held-out splits measure in-perspective and cross-perspective generalization.
  • Model: AudioLens-R1 directly processes raw audio and natural-language perspectives using reasoning distillation and preference optimization.The model generates structured clustering answers without a separate clustering algorithm.
  • Results: AudioLens-R1 improves overall ARI by 12.99 points and V-measure by 11.62 points over baselines.The reported gains support its effectiveness for perspective-conditioned audio clustering.

2 Benchmark Construction

AudioLens-Bench combines diverse speech domains with natural-language perspectives spanning linguistic and paralinguistic criteria. Its construction controls semantic evidence and acoustic attributes, while its splits test recombination, new-audio, and unseen-perspective generalization.

  • Benchmark goals: AudioLens-Bench evaluates clustering speech recordings according to flexible natural-language perspectives requiring linguistic and paralinguistic reasoning.The same collection may induce different valid partitions under different perspectives.
  • Source corpora: The benchmark draws on legal cases, financial disclosures, banking requests, and task-oriented dialogues.The source corpora are converted into speech while preserving clustering evidence.
  • Perspective construction: Construction induces candidate perspectives, refines their categories, and converts them into instructions that omit category names.This pipeline is illustrated in the benchmark-construction overview.
  • Audio synthesis: Speech instances are created through expanded queries, evidence-preserving summaries, or dialogue-native transcripts, then validated against their perspectives.Generated transcripts cannot mention perspective or category names or near-verbatim instructions.
  • Audio synthesis: Controlled synthesis injects emotion, speaker identity, speaker count, and background acoustic conditions.For linguistic perspectives these attributes are balanced; for paralinguistic perspectives the target acoustic attribute defines clustering.
  • Quality control: Three co-authors retain samples only after agreeing on perspective relevance, transcript accuracy, intelligibility, label correctness, and attribute perceptibility.Otherwise, samples are discarded and regenerated.
  • Benchmark organization: Held-in and held-out perspective splits distinguish familiar-criterion generalization from cross-perspective generalization.Evaluation levels L0, L1, and L2 respectively test recombination, new audio under familiar perspectives, and unseen perspectives with unseen audio.

3 Method

AudioLens-R1 performs perspective-conditioned clustering directly from indexed speech collections and natural-language perspectives, inferring valid partitions and the cluster count. It is trained first with comparison-based reasoning distillation and then with preference optimization over valid but incorrect partitions.

  • Model: AudioLens-R1 is an end-to-end large audio-language model for audio multi-perspective clustering.Its two training stages are reasoning distillation and preference optimization.
  • Problem formulation: The task maps an indexed audio collection and natural-language perspective to a partition of the recordings.The model must infer the number of clusters, and cluster names or order do not affect correctness.
  • Reasoning distillation: Reasoning distillation trains comparison-based behavior from teacher traces paired with gold partitions.The traces teach perspective interpretation, recording comparison, cluster-count inference, and structurally valid output.
  • Reasoning distillation: The distillation data includes complementary traces grounded in transcribed content for linguistic perspectives and audible cues for paralinguistic perspectives.Traces are filtered to match the gold partition and remain concise and modality-consistent.
  • Preference optimization: Direct Preference Optimization contrasts the canonical gold partition with parsable, valid but incorrect partitions sampled from the reasoning-distilled model.This focuses learning on clustering errors rather than formatting failures.
  • Preference optimization: Hard negatives include over-merging, over-splitting, wrong cluster counts, and wrong item assignments, balanced across perspectives and error types.The selection targets informative structural mistakes rather than a narrow error class.
  • Optimization objective: DPO optimizes final clustering answers, while reasoning traces provide intermediate supervision during distillation.The preference objective computes log-probabilities only over the canonical clustering answer.

4 Experimental Results

AudioLens-R1 outperforms ASR-based, embedding-based, and native audio-language baselines across the benchmark, with especially strong and balanced performance across clustering perspectives. Ablations show that reasoning distillation and preference optimization are complementary, while training also stabilizes the model’s response format and preference alignment.

  • Overall Results: AudioLens-R1 achieves the best overall ARI of 44.77 and V-measure of 73.43, improving over GPT-audio-1.5 by +12.99 and +11.62 absolute points, respectively.It achieves the best ARI in all clustering settings and the best V-measure in 11 of 12 settings.
  • Perspective-level Performance: AudioLens-R1 leads background-noise, emotion, and linguistic-reasoning perspectives under both ARI and V-measure.Its macroaveraged results are (41.45/60.35) for background noise, (39.00/69.00) for emotion, and (48.38/77.10) for linguistic reasoning.
  • Perspective-level Performance: AudioLens-R1 maintains a more balanced perspective profile than specialized baselines, with ARI ranging from 36.13 to 48.38 and V-measure from 54.25 to 77.10.Whisper+GPT-4o remains competitive for linguistic reasoning but performs substantially worse on background noise, emotion, and gender.
  • Ablation Study: DPO alone improves overall ARI from 34.83 to 37.07 and V-measure from 65.68 to 67.39, whereas reasoning distillation alone raises ARI to 35.97 with only a +0.28 V-measure gain.DPO alone still shows limited improvement in some cases, including MultiWOZ V-measure.
  • Ablation Study: Combining reasoning distillation and DPO yields 44.77 overall ARI and 73.43 overall V-measure, outperforming the baseline by +9.94 and +7.75 points.Adding reasoning distillation to DPO alone further improves ARI by +7.70 and V-measure by +6.04, indicating complementary effects.
  • Training Dynamics: Training rapidly stabilizes the required response format, with <think> tokens comprising most output and compact <answer> segments after an initial spike in extraneous tokens.DPO also increases the preference margin while gradually diverging from the reference model.

5 Related Work

Prior clustering methods rely on classical algorithms or learned representations, while LLM and audio-language approaches are beginning to support more flexible structure discovery. Audio-language models remain comparatively underexplored for organizing speech collections.

  • Classical clustering methods such as K-Means and GMMs partition data, while pretrained speech encoders provide richer acoustic representations.
  • Instruction-aware embedding models condition representations on natural-language instructions for clustering.
  • LLMs have been explored as clustering components, with reasoning-based methods framing clustering as a generative reasoning task.
  • Recent large audio-language models unify speech and text modeling, but prior work has focused mainly on recognition, generation, and audio understanding.

6 Conclusion

The paper introduces perspective-conditioned clustering for speech, a benchmark spanning linguistic and paralinguistic perspectives, and an end-to-end reasoning-trained audio-language model. AudioLens-R1 consistently outperforms baselines across the reported clustering metrics, supporting native audio-language models for flexible speech organization.

  • Audio multi-perspective clustering requires organizing speech recordings under a natural-language perspective while inferring cluster number and assignments.
  • AudioLens-Bench spans diverse application domains and combines linguistic with paralinguistic clustering perspectives.
  • AudioLens-R1 is an end-to-end large audio-language model trained with reasoning distillation and preference optimization.
  • AudioLens-R1 consistently outperforms baselines across ARI and V-measure.
  • Native large audio-language models may serve as flexible clustering agents for perspective-conditioned organization, retrieval, and analysis of audio data.

A Corpus-Specific Benchmark Construction

AudioLens-Bench is constructed by inducing and refining natural-language clustering perspectives, generating and validating transcripts, and synthesizing audio with controlled attributes. The pipeline targets interpretable, diverse perspectives while preserving labels needed for benchmark evaluation.

  • Benchmark construction: The benchmark uses “clustering perspective” for a natural-language criterion that groups audio recordings; generated data may call this internal concept “dimension.”
  • Benchmark construction: All corpora follow perspective induction, perspective refinement, transcript construction, transcript validation, and audio synthesis.
  • Perspective induction: An LLM proposes candidate perspectives from representative examples, targeting reasoning factors such as intent, interaction pattern, discourse function, risk, and consequence.
  • Perspective refinement: Perspective refinement removes vague, imbalanced, weakly supported, or shallow-lexical perspectives and enforces mutually exclusive, comparable categories.
  • Perspective refinement: Refined perspectives receive neutral instructions that begin with “Cluster” or “Group,” avoid taxonomy terms, and emphasize the dominant factor.
  • Transcript construction and validation: Generated transcripts preserve evidence for assigned labels, while independent classifier agreement filters label drift and explicit category or instruction mentions.
  • Audio synthesis: Audio synthesis converts validated transcripts to speech and injects controlled emotion, speaker, speaker-count, and background-acoustic attributes.

B.4 Episode Construction and Benchmark Statistics

Benchmark episodes pair a natural-language perspective with indexed audio clips and require models to infer both partition cardinality and assignments. Controlled sampling creates varied episode structures, while the training statistics show compact, mostly nonsingleton partitions with substantial source-specific complexity.

  • Episode formulation: Each episode contains a natural-language perspective and 4–10 indexed audio clips, with the target represented as a partition over those clips.
  • Episode formulation: The model receives neither semantic category names nor the gold cluster count and must assign every clip to exactly one cluster.
  • Controlled episode sampling: Held-in recordings use separate training and evaluation pools, while held-out perspectives are reserved for evaluating L2 generalization.
  • Controlled episode sampling: Quota-aware sampling balances feasible cluster counts, down-weights trivial single-cluster episodes, and varies clip counts conditionally on K.
  • Episode-level statistics: 1876 training episodes are constructed from 2008 unique recordings, with approximately 10.6K clip occurrences and an average reuse factor of 5.3×.
  • Episode-level statistics: Training episodes average 5.64 clips and 3.37 clusters, yielding approximately 1.67 clips per cluster and an average duration of 373.5 seconds.
  • Cluster structure: Singletons comprise 52.3% of gold clusters, but only 4.7% of episodes contain one cluster and 94.1% contain a nonsingleton cluster.
  • Source-specific complexity: MultiWOZ has denser partitions, whereas Banking77’s higher cluster cardinality mainly reflects more fine-grained categories rather than larger groups.

C Additional Method Details

AudioLens-R1 is trained through reasoning distillation and preference optimization to produce perspective-conditioned audio partitions. The procedure combines gold-constrained reasoning traces with hard-negative preference pairs and separate linguistic and paralinguistic supervision.

  • Reasoning Distillation: Teacher traces identify grouping principles, compare segments, resolve confusable cases, infer cluster counts, and assign every item exactly once.
  • Reasoning Distillation: Gold-constrained rationale synthesis conditions the teacher on the correct partition, separating answer discovery from explanation generation.
  • Linguistic and Paralinguistic Traces: Linguistic traces use transcribed indexed inputs, whereas paralinguistic traces give indexed audio directly to an audio-capable teacher model.
  • Reasoning Prompt Design: The prompts require group-level comparison, concise reasoning, and a recoverable final clustering that covers every item exactly once.
  • Preference Optimization: For preference optimization, candidates with extremely small clustering-quality gaps are removed, leaving valid but meaningfully worse partitions as negatives.

D Baseline Implementation Details

The baselines implement clustering through transcript embeddings, transcription-based language-model prompts, or direct native-audio prompting. Native-audio protocols preserve access to both linguistic content and relevant paralinguistic cues.

  • Transcript-Embedding Baselines: Transcript-embedding baselines transcribe each segment and cluster task-conditioned transcript representations using four specified embedding backbones.
  • Transcription-Based LLMs: Transcription-based language-model clustering uses ASR-generated scripts and prompts the model to cluster them according to the task goal.
  • Output Protocol: All baseline output protocols require every item to appear in exactly one cluster without omissions or duplication.
  • Native-Audio LALMs: Native-audio LALM baselines receive audio segments directly and listen to them in order under 1-based item indexing.

E Implementation Config Details

AudioLens-R1 is trained in two stages with parameter-efficient adaptation: reasoning distillation followed by scaled-mean DPO on screened preference pairs.

  • Reasoning Distillation: Reasoning distillation trains on 1,876 examples from a 7B Audio Flamingo 3 initialization with LoRA adaptation.
  • Preference Optimization: DPO uses 1,075 held-in-perspective training pairs, with both policy and reference initialized from the reasoning-distilled model.
  • Preference Optimization: The DPO run uses scaled-mean optimization with β = 0.5 and a fixed reference length scale Lref = 34.0.
  • Preference Optimization: DPO training runs for 500 steps with five warmup steps and a learning rate of 5 × 10−6.

F Evaluation Metrics

The evaluation combines V-measure and Adjusted Rand Index to assess complementary aspects of clustering quality. V-measure balances homogeneity and completeness, while ARI measures chance-adjusted pairwise consistency.

  • V-measure: Homogeneity reaches 1 when each cluster contains only one class.
  • V-measure: Completeness reaches 1 when all members of a class are assigned to one cluster.
  • V-measure: V-measure is the harmonic mean of homogeneity and completeness, reaching a high value only when both are simultaneously high.
  • Adjusted Rand Index: ARI applies chance-adjusted normalization to pairwise agreement, with 1 indicating perfect agreement and 0 corresponding to random assignments.
  • Metric Complementarity: Using both metrics evaluates semantic purity and structural consistency in audio multi-perspective clustering.

G More Analysis

AudioLens-R1 outperforms ASR-based and native audio-language baselines, with gains across corpora and evaluation settings. Its failures also show that reliable perspective-conditioned clustering requires task-specific behavior beyond general audio understanding.

  • Comparison with ASR-based Pipelines: AudioLens-R1 outperforms Whisper+GPT-4o by +16.39 ARI points while operating directly on audio inputs.Whisper+GPT-4o obtains 28.38 overall ARI and 60.42 overall V-measure, whereas Instructor with K-Means reaches 17.01 ARI and 52.66 V-measure.
  • Comparison with Native LALMs: AudioLens-R1 reaches 44.77 ARI and 73.43 V-measure, improving over GPT-audio-1.5 by +12.99 ARI points and +11.62 V-measure points.It achieves the best ARI across all 12 corpus-level settings and the best V-measure on 11 of 12 settings.
  • Failure Mode of Audio Flamingo 3: Audio Flamingo 3’s very low scores mainly reflect unreliable adherence to the required clustering-output protocol.Its responses often omit items, duplicate assignments, or lack explicit cluster structure.
  • Failure Mode of Audio Flamingo 3: Even after assignment extraction, Audio Flamingo 3 remained weak, indicating a task-specific deficiency rather than merely a parsing artifact.The required behavior includes interpreting the perspective, jointly comparing audio segments, inferring cluster count, and producing a complete partition.
  • Corpus-level Observations: AudioLens-R1’s gains are consistent across all four corpora, with especially strong ARI improvements on Banking77.The improvement patterns differ by domain: ECHR and S&P 500 show gains in both ARI and V-measure, while Banking77 emphasizes fine-grained intent distinctions.

G.1 Case Study

The ECHR case study tests whether AudioLens-R1 can cluster legal speech by the relationship between applicants and responsible actors. The model forms four relation-based groups that follow the requested perspective rather than superficial topic similarity.

  • G.1 Case Study: The case study asks AudioLens-R1 to cluster legal cases by the main relationship between applicants and responsible actors.The perspective distinguishes direct State conduct from protection, enforcement, or regulatory failures, while inference uses original audio recordings.
  • G.1 Case Study: AudioLens-R1 separates seven cases into four relation-based groups: administrative or regulatory disputes, direct coercive State conduct, private judicial-protection disputes, and detention-condition complaints.The grouping is reported in Fig. 7.
  • G.1 Case Study: Police abuse and military police shooting are grouped as direct State force, while customs seizure, land-transfer approval, and contaminated blood-product compensation form an administrative or regulatory group.These examples illustrate that the partition follows responsibility relations rather than shared surface topics.
  • G.1 Case Study: The case study suggests that AudioLens-R1 can infer abstract relational structures from native audio inputs under user-specified perspectives.This conclusion concerns the demonstrated ECHR case study.
Loading 2608.25177v1…