PaperlayerView exact revision on arXiv

Source-linked AI summary

Don't Just Listen, Try Planning: Graph-based Retrieval-Generation Agent for Long-form Audio Meeting Understanding

Quanwei Tang, Dong Zhang, Shoushan Li, Guodong Zhou

arXiv:2608.24048v1cs.SDcs.AI

TL;DR

Long-form audio meeting QA lacks dedicated datasets, and existing approaches lose acoustic information or struggle with long-range context. The paper introduces LongAudioQA and GRGA, which organize heterogeneous audio into a graph and plan retrieval and generation. GRGA outperforms end-to-end Speech LLMs and RAG-based systems, while upstream noise and inference cost remain important boundaries.

  • Problem

    LongAudioQA addresses the scarcity of dedicated QA datasets for long-form meetings and the difficulty of preserving acoustic information and long-term context.

  • Method

    GRGA models heterogeneous audio features in a multimodal graph and formulates QA as planned, reflective retrieval and answer generation.

  • Results

    GRGA significantly outperforms end-to-end Speech LLMs and RAG-based state-of-the-art methods on LongAudioQA benchmarks.

  • Takeaways & Limitations

    Graph-based planning supports long-form QA by explicitly navigating and reasoning over semantic, temporal, speaker, and acoustic relationships.

  • Takeaways & Limitations

    Graph construction depends on upstream ASR and diarization, while iterative planning and reflection increase computational cost and limit real-time deployment.

Abstract

from arXiv · show

While long-form audio meeting understanding (LAMU) is garnering growing attention, task-specific question answering (QA) datasets remain scarce. Existing speech QA paradigms and state-of-the-art Speech LLMs suffer from acoustic information loss and poor long-term context memory. To address these issues, we construct the LongAudioQA dataset and propose the GRGA model, which models heterogeneous audio features into a multi-dimensional graph and leverages agent planning for retrieval and answer generation.

1 Introduction

LAMU lacks dedicated QA datasets for long, multi-party meetings, while existing speech QA and Speech LLM approaches lose acoustic information and long-range context. LongAudioQA and GRGA address these gaps by representing heterogeneous audio features in a multi-dimensional graph and planning retrieval and answer generation.

  • LAMU research has focused mainly on transcription recognition, leaving dedicated QA datasets for long-form multi-party meetings scarce.LongAudioQA is designed around complex semantics, multi-speaker interactions, and long timestamps.
  • Existing Speech LLMs prioritize ASR-derived textual context, which loses valuable acoustic information such as voice and tone.The paper illustrates this limitation with questions about a sudden loud voice at a specific timestamp.
  • Short-form speech QA methods and standard RAG approaches fail to capture long-term dependencies and can fragment the rationale across distant timestamps.The motivating example requires linking content at 19 minutes to an earlier description at 1:30.
  • GRGA models acoustic information and speaker attributes in a unified multi-dimensional graph, then uses agent planning to retrieve relevant clues and generate answers.The graph includes heterogeneous features such as voice, tone, speaker role, and gender.
  • LongAudioQA and GRGA are the paper’s central contributions, supported by automatic and human evaluation on three datasets.The evaluation is conducted on the proposed LongAudioQA benchmarks.

2 Dataset Construction

LongAudioQA is constructed to test multi-hop, temporal, inferential, and acoustic-aware reasoning over complex meeting interactions rather than flat text retrieval. It draws from diverse speech corpora and applies expert verification to improve question and timestamp-evidence reliability.

  • Existing benchmarks emphasize short-context extraction or simple intent classification, leaving long-form multi-hop reasoning and temporal grounding under-tested.The dataset targets interactions involving speaker turns, cross-references, and temporal dynamics.
  • LongAudioQA uses AliMeeting, AMI, and DailyTalk to cover overlapping multi-speaker audio, acoustically challenging meetings, and cleaner dyadic conversations.DailyTalk contains 2,541 dialogues and is concatenated into inputs shorter than 10 minutes.
  • Its question taxonomy includes factual, inferential, temporal, summarization, and acoustic-aware categories.Acoustic-aware questions require grounding textual semantics with paralinguistic acoustic signals.
  • The dataset evaluates retrieval from global long-context conference data without prior knowledge of relevant segments, directly targeting context fragmentation.This setup tests whether models can locate answers across long recordings.
  • Expert verification removes hallucinated questions, rewrites ambiguous references, validates reasoning depth, and requires timestamp evidence IoU above 0.9 with ground truth.The resulting dataset achieves κ = 0.91 Cohen’s Kappa inter-annotator agreement.

3 Methodology

GRGA treats long-form meeting QA as a planning problem: it represents heterogeneous audio and dialogue relations in a multi-dimensional graph, then searches, synthesizes, and verifies evidence through an agent loop.

  • Overall Framework: GRGA models long-form audio as a multi-dimensional graph capturing semantic, temporal, and speaker dependencies.The graph includes acoustic information and speaker attributes alongside transcript-derived content.
  • Audio and Speaker Processing: The pipeline segments audio, aligns transcripts with acoustic features, and profiles speakers to initialize semantically meaningful graph nodes.Speaker profiling aggregates utterances and stores attributes such as role, gender, and stance as node information.
  • Graph Construction: The graph represents each utterance as a node enriched with transcript, speaker identity, timestamps, and corresponding speech.Multiple edge types connect adjacent utterances, conversational replies, same-speaker utterances, entities, and semantic coreference relations.
  • Planning Process: GRGA decomposes a query into entity, concept, time, and metadata constraints before planning graph-retrieval operations.Its execution planner produces a sequence of atomic actions, enabling chained operations such as Search → Filter.
  • Planning Process: The execution engine applies each plan to the graph and returns partial observations that update the agent’s belief state.The process is formulated as interaction with a graph environment in a partially observable decision process.
  • Planning Process: Answer synthesis proposes a response from accumulated evidence, while reflection verifies entailment and either terminates or triggers replanning.A successful verification receives a positive reward; failure injects critique into the belief state for corrective planning.

4 Experimentation

The experiments evaluate GRGA against end-to-end Speech LLMs and RAG baselines across long-form meeting datasets, using semantic and human-centered measures. Results indicate stronger long-context reasoning, grounding, citation quality, and robustness to upstream noise.

  • Experimental Setup: Experiments compare GRGA with competitive multimodal LLMs using whole-meeting speech or transcription contexts, plus TextRAG and AudioRAG baselines.The evaluation uses LongAudioQA and reports accuracy across three datasets and question types.
  • Evaluation Metrics: Semantic Accuracy uses an LLM judge to assess semantic equivalence, factual consistency, and reasoning correctness beyond lexical overlap.The judge disregards stylistic differences and evaluates whether generated answers entail the same semantic information as the ground truth.
  • Main Results: AudioFlamingo3 drops from ∼65% on DailyTalk to ∼16% on AMI, while GRGA outperforms MiMo-Audio on AMI.The comparison is presented as evidence that fixed context limits hinder long-form audio reasoning while graph-based retrieval scales beyond them.
  • Main Results: TextRAG scores 24.6% versus GRGA’s 65.3% on inferential AMI questions, while AudioRAG remains near ∼20%.The authors attribute the gap to multi-hop dependencies and acoustic retrieval noise, motivating query planning and a structured graph intermediate.
  • Human Evaluation: GRGA outperforms Qwen3-Omni across datasets and metrics, with a Groundedness gap of +1.27 on AllMeeting.The authors connect stronger grounding with higher Correctness through precise temporal anchoring.
  • Robustness Analysis: GRGA maintains high QA accuracy despite AMI and AliMeeting transcription and diarization noise, including WERs near 20% and DERs above 13%.The analysis associates this robustness with iterative planning and reflection mechanisms.
  • Evidence Analysis: Citation Precision improves over standard Text RAG by +19.3% on AMI and +6.7% on AliMeeting.The authors associate these gains with query planning and reflection filtering irrelevant or hallucinated citations.
  • Ablation Study: Removing any GRGA module degrades performance, while removing Graph Traversal highlights the importance of the multi-dimensional graph.The ablation is conducted on AMI, with the table reporting accuracy drops relative to the full framework.

5 Related Work

Related work covers speech and meeting QA, large Speech LLMs, retrieval-augmented generation, and tool-using agents. These lines of work leave long-context, multi-hop, and structured audio understanding insufficiently addressed.

  • Speech and Meeting Question Answering: Early speech QA focused on answer-span extraction or ranking from short, single-speaker segments rather than generative meeting QA.Meeting datasets such as AMI and AliMeeting are mainly used for ASR and diarization.
  • Large Speech Language Models: Speech LLMs process audio directly but face context-length limits; AudioFlamingo3 is typically constrained to approximately 10 minutes.Kimi-Audio is optimized for clips shorter than 30 seconds and suffers truncation on longer inputs.
  • Retrieval-Augmented Generation: Traditional one-shot RAG retrieves contexts through dense vector similarity but struggles when evidence is fragmented or requires multi-hop reasoning.Semantic similarity alone does not capture all logical deduction steps needed by such questions.
  • LLM Agents and Tool Using: Tool-using agents have shown proficiency in open-domain tasks, while their application to structured audio understanding remains underexplored.The paper positions its meeting-specific action space as a bridge between these areas.

6 Conclusion

The paper presents GRGA to address acoustic missing and context forgetting in long-form speech understanding. It structures audio as a multimodal heterogeneous graph and frames QA as a POMDP for explicit planning, navigation, and reasoning, with experiments showing superiority over Speech-LLM and RAG baselines.

  • Conclusion: GRGA addresses acoustic missing and context forgetting by structuring audio into a multimodal heterogeneous graph.The framework models long-form speech understanding through explicit graph structure.
  • Conclusion: GRGA formulates question answering as a POMDP, enabling an agent to plan, navigate, and reason over complex interactions.This agentic formulation is the paper’s core mechanism for handling long-form meeting content.
  • Conclusion: Experiments on LongAudioQA benchmarks show that GRGA significantly outperforms both end-to-end Speech-LLMs and RAG-based SOTAs.The conclusion reports this benchmark-level comparison as the principal outcome.

Limitations

The paper identifies three limitations: vulnerability to upstream audio-processing errors, higher inference latency, and evaluation restricted to structured meetings.

  • Graph construction can propagate severe acoustic-noise or speaker-overlap artifacts from upstream ASR and diarization into reasoning.
  • Iterative planning and reflection cost more computationally than single-turn RAG, limiting real-time deployment.
  • The evaluation focuses on structured meetings, leaving generalization to unstructured domains such as movies or vlogs for future work.

Ethics Statement

The dataset uses licensed public corpora and human annotation safeguards, while acknowledging risks from LLM-generated data and workplace surveillance.

  • The dataset draws on AliMeeting, AMI Meeting Corpus, and DailyTalk under their original licensing terms, with review for unconsented PII exposure.
  • Graduate-student annotators were paid above the approximately $5-per-hour local minimum wage and protected from potentially harmful content.
  • LLM-generated questions were checked by humans for factual consistency, stereotypes, and hallucinated events, achieving κ = 0.95 agreement.
  • The authors identify dual-use risks including unauthorized workplace surveillance and privacy intrusion, recommending consent and robust encryption.

A More Analysis in Ablation Study

Ablation results show that planning, reflection, graph traversal, audio access, and semantic search each support different aspects of the system’s performance.

  • Impact of Cognitive Modules: Removing the Query Planner drops performance by 4.80%, including an 8.94% decline on inferential tasks.
  • Impact of Cognitive Modules: Removing Reflection causes a 9.84% decline, consistent with the role of verification in rejecting irrelevant retrieved chunks.
  • Impact of Graph Tools: Disabling Graph Traversal produces a ∆= −11.27% degradation and prevents aggregation through speaker and temporal edges.
  • Impact of Graph Tools: Removing Audio Access reduces acoustic-aware question accuracy from 35.68% to 12.06%, showing that transcripts alone miss paralinguistic cues.
  • Impact of Graph Tools: Removing Semantic Search causes a ∆= −33.21% collapse because the agent loses its entry point for locating evidence nodes.
  • Across three datasets, the method consistently outperforms the vanilla model and RAG baselines on evidence citation accuracy.

C Step Analysis

The agent adapts reasoning depth and tool usage to question complexity: factual queries are handled efficiently, while inferential, temporal, and acoustic-aware queries receive deeper or audio-based processing.

  • Figure 6 depicts reasoning steps by question type and stacked tool-usage distributions, highlighting heavier Graph Traversal use for multi-hop reasoning.
  • Complexity-Aware Reasoning: Factual queries are typically resolved in just 1.8 reasoning steps using primarily Semantic Search.
  • Complexity-Aware Reasoning: Inferential and Temporal queries average more than 4 steps, involving iterative traversal of temporal or speaker edges.
  • Tool Usage Distribution: Summarization tasks predominantly use Filter tools to isolate specific time ranges or speakers.
  • Tool Usage Distribution: Acoustic-aware questions frequently invoke Audio Access late in the process when transcripts cannot resolve cues such as emotion.

D Human Evaluation.

Blind human evaluation across 150 queries shows that GRGA produces more grounded and correct answers, with its advantages strongest on noisy, multi-party meetings.

  • GRGA improves Groundedness by +0.97 over Qwen3-Omni, whose high Coherence score of 4.07 coexists with low Groundedness of 2.69.The comparison indicates a gap between fluent generation and factual anchoring.
  • GRGA anchors answers to precise timestamps and speakers, mitigating hallucinations during human evaluation.
  • GRGA gains +0.52 in Correctness by explicitly locating evidence, especially on reasoning-heavy questions.The paper reports a positive correlation between groundedness and correctness.
  • Figure 7 compares QA Accuracy under varying upstream error rates and highlights a widening ∆Accuracy gap for GRGA in noisier environments.
  • GRGA’s advantage is most pronounced on noisy, multi-party AMI and AliMeeting meetings, including a +1.13 Groundedness gain on AMI.The method retains advantages over standard omni-models in speaker attribution and temporal reasoning.

E Noise Sensitivity Analysis

The analysis evaluates GRGA and Text RAG across increasing ASR and diarization noise. GRGA degrades more gracefully, with its advantage widening under severe corruption and reflection-enabled re-planning improving answer grounding.

  • Evaluation Setup: Robustness is evaluated across a 3 × 3 grid combining simulated Word Error Rate and Diarization Error Rate conditions for GRGA and Text RAG.The analysis varies upstream transcription and speaker-label reliability to compare both systems under matched noise settings.
  • Noise Conditions: Three levels define each noise dimension: Low uses ground-truth transcripts and labels, Mid uses the standard pipeline, and High introduces simulated severe errors.The standard condition has DER ≈17.6% and WER ≈18.8%; High conditions reach WER ≈33.6% and DER ≈52.4%.
  • Results: 54.47% to 24.67%: Text RAG’s accuracy drops sharply from Low/Low to High/High, while GRGA retains 39.54% under the harshest conditions.The comparison characterizes standard retrieval as fragile when transcription or speaker cues become noisy.
  • Results: +5.37% to +14.87%: GRGA’s performance gap over Text RAG widens from Low/Low to High/High, consistent with a noise-buffer effect.The paper attributes this behavior to topological connectivity and temporal constraints that reduce reliance on perfect transcription and diarization.
  • Self-Correction: The reflection module can trigger re-planning after a low-confidence response, directing retrieval toward specific evidence such as concrete deadline complaints.In the case study, this changes the answer from generic and low-confidence to highly grounded and accurate.
  • Evaluation Metric: The evaluation reports Semantic Accuracy, using an LLM judge to assess entailment and factual reasoning while ignoring stylistic differences in generated responses.This metric is intended to validate retrieval and reasoning over core knowledge beyond rigid lexical matching.
Loading 2608.24048v1…