PaperlayerView exact revision on arXiv

Source-linked AI summary

AudioSpan: Spanning the Duration and Depth of Audio Comprehension

Wen Huang, Yunfei Chu, Meng Gao, Haolin He, Jin Xu

arXiv:2608.26431v1cs.SDeess.AS

TL;DR

Existing audio-language benchmarks mostly test seconds-long clips, limiting evidence about sustained comprehension over long, redundant audio. AudioSpan evaluates 12 models with long-form audio across perception, understanding, and reasoning, finding that extracting relevant facts before reasoning is the main bottleneck and worsens with duration.

  • Problem

    Existing LALM benchmarks mainly use seconds-long clips, leaving long-context audio comprehension insufficiently evaluated.

  • Method

    AudioSpan pairs minutes-to-hours in-the-wild audio with three cognitive levels and complementary Native QA and Anchor QA question paths.

  • Results

    Across 12 LALMs, distilling relevant facts from long, redundant audio bottlenecks performance before reasoning, with perception weakest and difficulty increasing with duration.

  • Takeaways & Limitations

    AudioSpan provides a baseline for tracking whether models can sustain grounded comprehension across longer audio spans.

  • Takeaways & Limitations

    Native QA depends on caption fidelity, while Anchor QA restricts question diversity; the benchmark covers only English and Chinese in-the-wild speech.

Abstract

from arXiv · show

General audio comprehension now covers speech, sound, and music over durations from seconds to hours, driven by large audio-language models (LALMs) that are increasingly omni-modal. Yet the benchmarks that test them still rely on clips of seconds, where scores saturate and models converge; recent long-form efforts extend duration but evaluate long audio much as short clips are. We introduce AudioSpan, a benchmark that spans both duration and depth: it pairs audio from 10 minutes to over 2 hours with 3,240 questions across three cognitive levels, namely perception, understanding, and reasoning. Two paths supply the questions, differing in how question content is sourced and how ground truth is obtained. Native QA extracts questions from the audio's content, posing each as a multiple-choice item and an open-ended one graded by detailed rubrics. Anchor QA instead injects ground truth, planting acoustic anchors into the audio and building a perception-to-reasoning chain scored only to the first error. A fully automated pipeline constructs every item through structured captioning, QA generation, and adversarial critic feedback. Evaluating 12 LALMs on AudioSpan, we find the hard part comes before reasoning: distilling a few relevant facts from a long, redundant signal. This difficulty grows with audio length and falls hardest on perception, especially temporal grounding. AudioSpan is available at https://huggingface.co/datasets/holvan/AudioSpan.

1 INTRODUCTION

AudioSpan addresses limitations in existing audio-comprehension benchmarks by spanning minutes to hours and three cognitive levels with dual evaluation paths and automated quality-controlled construction. Evaluations of 12 LALMs show that extracting relevant facts from long, redundant audio is the primary bottleneck, worsening with audio length.

  • Motivation: Top benchmark accuracy rose from around 50% to over 80% within a year, while leading LALMs now cluster within a few points.Existing benchmarks use short audio clips, where scores increasingly saturate and models converge.
  • Motivation: Existing long-form benchmarks remain limited in task frameworks, question formats, data construction, and quality assurance.They often use fixed schemes, multiple-choice or fixed-answer questions, synthetic or corpus-derived audio, costly annotation, or caption-dependent pipelines.
  • AudioSpan: AudioSpan spans minutes to hours and perception, understanding, and reasoning through a fully automated, scalable pipeline with structured captioning, QA generation, quality control, and adaptive refinement.The benchmark collects long-form audio in the wild and uses dual-path evaluation across three cognitive levels and three question formats.
  • Findings: 12 recent LALMs are bottlenecked by distilling a few relevant facts from long, redundant audio before reasoning, and this difficulty grows with audio length.The evaluation covers seven open-source and five proprietary models.

2 RELATED WORK

Audio-language modeling has evolved from modality-specific systems toward unified models spanning speech, environmental sound, and music, while open releases increasingly include audio-only specialists. Existing benchmarks remain largely short-clip evaluations, and audio QA construction and scoring balance scalability, grounding, determinism, and comprehension fidelity.

  • Audio-language models: Audio-language modeling progressed from modality-specific systems to unified models covering speech, environmental sound, and music.Early work paired general audio encoders with language models for captioning and question answering over short clips.
  • Audio-language models: Open audio-language model releases have proliferated, including audio-only specialists such as Audio Flamingo, Voxtral, Kimi-Audio, MiMo-Audio, StepAudio, Eureka-Audio, and MOSS-Audio.
  • Audio benchmarks: Benchmarks initially focused on clips under one minute, covering perception and understanding across speech, sound, and music, with some later targeting deeper reasoning.MMAU-Pro broadens task coverage and adds open-ended items, while MMAR, MMSU, and STAR-Bench target deeper reasoning on short clips.
  • Audio QA construction: Expert annotation provides reliable audio-grounded answers but scales poorly, motivating caption-then-QA pipelines that derive questions from textual descriptions.
  • Evaluation: Multiple-choice audio benchmarks enable deterministic grading with a 25% random baseline for four options but may reward option elimination over genuine comprehension.

3 BENCHMARK DESIGN

AudioSpan’s benchmark design pairs Native QA, which derives questions from structured audio captions, with Anchor QA, which injects acoustic anchors and builds questions around their context. Both paths use a shared perception–understanding–reasoning taxonomy but differ in formats and scoring, producing 3,240 items from 360 multilingual, multi-duration audio samples.

  • Question paths: Native QA grounds audio-specific questions in structured captions, but caption omissions limit which content can become question ground truth, especially for hour-long audio.This dependency motivates the complementary Anchor QA path.
  • Question paths: Anchor QA injects ground truth by planting acoustic anchors into audio and constructing questions around their surrounding context.The operation itself defines the ground truth used for the questions.
  • Dataset composition: 3,240 items cover 360 audio samples balanced across English and Chinese and duration tiers from 10 minutes to over 2 hours.The item set contains 1,080 Native MCQs, 1,080 Native OEQs, and 1,080 Anchor MCQs organized as 360 three-question chains.
  • Cognitive taxonomy: The shared taxonomy progresses from Perception of explicit information, through Understanding of coherent meaning, to Reasoning beyond the given information.The levels represent information acquisition, interpretation, and inferential judgment.
  • Formats and scoring: Native questions are posed as four-option MCQs scored by exact match and OEQs graded by rubric-based LLM judges using content and, when needed, temporal criteria.OEQ criteria assign semantic scores of 0, 0.5, or 1 and temporal scores of 0 or 1 when time-bound.

4 DATA CONSTRUCTION

AudioSpan is constructed by an automated two-phase pipeline that captions each audio and generates Native or Anchor QA items. Automatic gates, solver-based checks, critic feedback, evidence verification, and human review control item quality.

  • Pipeline: The two-phase pipeline first converts audio into structured captions, then generates Native QA from content or Anchor QA from planted acoustic anchors.Native QA verifies ground truth against the audio, whereas Anchor QA reads ground truth from anchor settings.
  • Captioning: The caption pipeline combines timestamped speech transcription, semantic segmentation into coherent segments, and acoustic captioning.Segments are capped at about 5 minutes and annotated with titles, summaries, and key topics.
  • Native QA: Native QA selects question directions and relevant segments, then generates questions spanning perception, understanding, and reasoning from their captions.Retained multiple-choice questions are recast as open-ended questions with essential and supporting rubric criteria.
  • Anchor QA: Anchor QA selects 1–3 semantically rich anchor positions, modifies the audio to establish ground truth, and generates a three-level question chain from anchor context and related segments.It reuses four Native QA steps, but applies verification and refinement only to understanding and reasoning and omits rubric generation.
  • Quality assurance: A candidate is kept only when blind and text-only solvers fail while the full solver succeeds, after which critic feedback, evidence checks, and human review further filter items.Evidence fragments are validated against the audio, critic lessons are stored in separate Native and Anchor memory banks, and reviewers can accept, fix, or reject items.

5 EXPERIMENTS

Experiments evaluate 12 native-audio models against text-only baselines across three modes and duration tiers. Results show that long, redundant audio burdens comprehension before reasoning, with performance worsening with duration and perception—especially grounded, time-sensitive perception—being weakest.

  • Experimental setup: 12 models are evaluated under a native-audio protocol, with audio truncated at each model’s input limit when necessary and never replaced by transcripts.Seven models are open-source and five are closed; the open-source models span audio specialists and omni-modal systems, while all closed models are omni-modal.
  • Experimental setup: Three GPT-5.4 text-only baselines provide increasing textual access: question-only, transcript, and the dense authoring caption.Question-only supplies a floor from answer-choice priors, transcript reads spoken words, and caption references the material used to build the benchmark.
  • Overall performance: The caption baseline leads every audio model on Accuracy and Rubric, indicating that the main difficulty lies in distilling relevant facts from the waveform before reasoning.The comparison holds content constant while changing modality: the text solver reads facts directly, whereas audio models must recover them from a long, redundant signal.
  • Duration effects: 6–11 accuracy points are lost from S to L by the transcript and dense-caption baselines, while question-only remains flat.Accuracy, Rubric, and Chain all decline from S to L, and the L tier, averaging approximately 90 minutes, is lowest for nearly every configuration.
  • Cognitive levels: Perception is the weakest cognitive level, with its deficit widening under Rubric and duration because time-sensitive perception is especially difficult.Across closed-source models, perception is lowest on both question paths; content recognition keeps pace with higher levels, but temporal grounding does not.

6 CONCLUSION

AudioSpan benchmarks grounded, long-form audio comprehension across minutes-to-hours durations and three cognitive levels using matched multiple-choice and open-ended formats. Across 12 LALMs, reliable grounding fails before reasoning and worsens with longer audio.

  • AudioSpan pairs minutes-to-hours audio with three cognitive levels and matched multiple-choice and open-ended scoring formats.It is designed for grounded, long-form audio comprehension.
  • Across 12 LALMs, models cannot reliably ground their descriptions in the audio before reasoning is tested.
  • The grounding gap widens as audio lengthens, making AudioSpan a baseline for tracking sustained-comprehension progress.

AI USAGE STATEMENT · APPENDIX CONTENTS

The paper uses generative AI both to construct AudioSpan and to assist manuscript preparation. Automated outputs undergo verification and human review before entering the benchmark.

  • AI USAGE STATEMENT: Large language model APIs generate structured captions, questions, and adversarial critic feedback for AudioSpan’s automated construction pipeline.These components are described in §4 and Appendix B.
  • AI USAGE STATEMENT: An LLM judge scores open-ended responses in the benchmark pipeline.This scoring component is described in §3.2 and Appendix C.
  • AI USAGE STATEMENT: All automated pipeline output passes verification and human review before entering the benchmark.The paper identifies these safeguards as part of the process described in the referenced sections and appendices.
  • AI USAGE STATEMENT: Generative AI tools were used as components of the research method.The statement explicitly characterizes AI tools as methodological components rather than only writing aids.
  • AI USAGE STATEMENT: The authors also used generative AI tools for manuscript preparation.The passage separately identifies manuscript preparation as an additional use.
  • APPENDIX CONTENTS: The AI usage statement points readers to §4, §3.2, Appendix B, and Appendix C for implementation and evaluation details.These references respectively cover pipeline construction and open-ended response scoring.

A DATA ETHICS … ENVIRONMENT

AudioSpan is released as a non-commercial evaluation resource with safeguards for copyright, privacy, and responsible use. Its construction pipeline generates Native and Anchor QA from structured, evidence-grounded captions, using global and segment-level context plus localized acoustic perturbations.

  • A.1 LICENSING AND RELEASE: AudioSpan’s artifacts are released under CC BY-NC-SA 4.0 for non-commercial research and evaluation, while source-recording copyrights remain with their original creators.The recordings are publicly available media included only for evaluation, and downloading the dataset entails agreement not to redistribute the audio.
  • A.2 ETHICS AND RESPONSIBLE USE: AudioSpan references speakers by role, excludes personal information from answer keys, and is not intended for speaker-identification or surveillance training.Human reviewers worked only with publicly available media and were compensated at or above local market rates.
  • GLOBAL: Figure 5 summarizes each audio’s topic, narrative arc, tone, speaker registry, segment index, and cross-segment dependencies for question construction.This audio-level overview drives direction selection for Native QA and anchor selection for Anchor QA.
  • B.1 CAPTION PIPELINE: Every question is generated from a structured caption rather than raw audio, combining an audio-level summary view with dense per-segment detail.The summary view supplies global context and cross-segment dependencies, while the dense view contains segment-specific information.
  • B.1 CAPTION PIPELINE: Caption evidence records assign stable IDs, segments, time ranges, modalities, and grounded content, which generation cites and verification re-checks against the audio.The records serve as the grounding unit throughout the construction process.
  • B.2 NATIVE QA PIPELINE: Native QA directions vary by cognitive level and dimension, with the generator selecting or combining directions for the audio at hand.Table 4 organizes the available directions by cognitive level and dimension under the free scheme of §3.2.
  • B.3 ANCHOR QA PIPELINE: Anchor QA applies one addition, deletion, and modification before question writing, each localized in time and layered across perception, understanding, and reasoning.Each operation is applied at one to three positions, enabling questions about trace counts and locations.
  • ENVIRONMENT: Anchor perturbations include 62 synthesized sound-event categories, obscured spans, and unchanged-word acoustic modifications in loudness or playback speed.Deletion uses silence or low-pass smearing near 300 Hz, while modification changes loudness by 8–12 dB or speed by a factor of 1.5–3.

B.4 QUALITY ASSURANCE · C EVALUATION PROTOCOL

AudioSpan’s quality assurance filters candidates for genuine audio dependence using multi-rollout automatic gates, adversarial critic feedback, and item-level human review. The gates retain roughly 8% of candidates, removing shortcut-solvable and evidentially unsupported questions before evaluation.

  • B.4 QUALITY ASSURANCE: Three independent solver rollouts gate each item, rejecting blind-solvable items above 1/3 correctness and text-only-solvable items above 2/3.The blind gate is stricter because the stem and options alone should almost never suffice.
  • B.4 QUALITY ASSURANCE: Questions must require audio-dependent evidence such as tone, delivery, timing, emphasis, or cross-segment integration.Invalid items include those whose answers are already stated, paraphrased, or causally explained by the transcript; distractors should remain plausible and transcript-grounded.
  • B.4 QUALITY ASSURANCE: Critic feedback is stored in a cognitive-level memory bank so later generation avoids specific recurring failure patterns.Figure 8 presents verbatim lessons from this bank grouped by cognitive level.
  • B.4 QUALITY ASSURANCE: Roughly 8% of candidates survive evidence verification and solver filtering.Most solver rejections are shortcut-solvable and are removed by the blind or text-only level.
  • B.4 QUALITY ASSURANCE: Perception items are most often rejected for citing events absent from the audio, while understanding items are most often rejected at the text-only level.The gates therefore expose different failure modes across cognitive levels.
  • B.4 QUALITY ASSURANCE: Human reviewers inspect each item against the relevant audio stretch, checking the open-ended question before the multiple-choice item.They use reference answers, rubric criteria, marked answers, and timestamps, and may accept an item or apply a minor wording fix.

C.1 EVALUATION SETUP … D.1 PERFORMANCE ACROSS LANGUAGES

AudioSpan evaluates models with audio-only inputs under fixed duration limits and scores responses through separate semantic and temporal criteria. The reported language results show that performance gaps depend on both model and answer format, with English generally stronger and Chinese performance varying substantially.

  • C.1 EVALUATION SETUP: Models receive native audio only, truncated to each model’s fixed limit using the leading prefix rather than answer-centered segments.If a model cannot process the full clip, it receives no transcript fallback; its limit is selected from stated documentation or trained-context capacity.
  • C.1 EVALUATION SETUP: Multiple-choice responses must be a single option letter, whereas open-ended responses must answer with audio timestamps for rubric evaluation.The two fixed instructions are respectively “Answer with ONLY the option letter” and “Cite audio timestamps in your analysis.”
  • C.2 SCORING DETAILS: Each criterion receives an importance weight and semantic score σc ∈{0, 0.5, 1} plus a temporal score τc ∈{0, 1}, with timestamps judged under ±5 s tolerance.Essential criteria use one applicable axis, while supporting criteria may combine semantic and temporal evidence through a gated score.
  • C.2 SCORING DETAILS: Semantic% excludes criteria without demanded claims, while temporal% excludes criteria without demanded timing and omitted timestamps.This separates content correctness from localization performance conditional on an attempted timestamp.
  • C.3 JUDGE PROMPT: The judge ensemble independently returns one semantic/temporal verdict pair per criterion after receiving the question, criteria, and model response.The prompt selects dimensions by criterion importance and content, normalizes timestamp formats, and evaluates semantic and temporal correctness independently.
  • C.4 MODELS AND APIS: Audio-facing construction stages use audio-capable models, while question generation and judging use text-only language models.The release’s construction-pipeline models are listed in Table 8.
  • D.1 PERFORMANCE ACROSS LANGUAGES: Most models achieve higher multiple-choice accuracy in English, but the language-gap magnitude and direction vary by model.Open models show larger English advantages in open-ended generation; Phi-4-Multimodal is near zero on Chinese Rubric, while the Chinese-origin Qwen-Omni family differs.

D.2 PERFORMANCE ACROSS DIMENSIONS · D.3 QUESTION FORMATS: MULTIPLE-CHOICE VS. OPEN-ENDED · D.4 RUBRIC AXES: SEMANTIC VS. TEMPORAL

Across dimensions, perception is weakest, especially temporal localization and counting, while multiple-choice often overstates grounded comprehension. Open-ended performance is constrained mainly by temporal grounding, which worsens with audio length despite comparatively stronger semantic accuracy.

  • D.2 PERFORMANCE ACROSS DIMENSIONS: D.2 PERFORMANCE ACROSS DIMENSIONS — 63.8% Accuracy and 43.6% Rubric: perception scores below understanding and reasoning under both metrics.Recognition keeps pace with the stronger levels, whereas localization and counting are the two lowest dimensions.
  • D.2 PERFORMANCE ACROSS DIMENSIONS: D.2 PERFORMANCE ACROSS DIMENSIONS — Localization retains only 53% of its Accuracy under open-ended grading, while counting remains low and barely declines under Rubric.Localization is most overstated by multiple choice because options supply timestamps; counting is a genuine performance cap.
  • D.3 QUESTION FORMATS: MULTIPLE-CHOICE VS. OPEN-ENDED: D.3 QUESTION FORMATS: MULTIPLE-CHOICE VS. OPEN-ENDED — Item-matched native questions compare each item’s MCQ correctness with its open-ended Rubric score and summarize their separation using AUC.AUC is 0.5 when MCQ correctness is uninformative and 1 when every correct item outscores every wrong one.
  • D.3 QUESTION FORMATS: MULTIPLE-CHOICE VS. OPEN-ENDED: D.3 QUESTION FORMATS: MULTIPLE-CHOICE VS. OPEN-ENDED — 53.8 Rubric+ versus 34.1 Rubric− yields only 0.65 AUC, staying within 0.62–0.66 across cognitive levels.The multiple-choice outcome is therefore barely above the 0.5 floor as a separator of open-ended quality.
  • D.3 QUESTION FORMATS: MULTIPLE-CHOICE VS. OPEN-ENDED: D.3 QUESTION FORMATS: MULTIPLE-CHOICE VS. OPEN-ENDED — 21% of matched pairs pass both formats, 15% fail both, 19% pass only multiple choice, and 6% pass only open-ended.About 39% of pairs fall into the partial middle; the multiple-choice-only cell outnumbers its mirror roughly three to one.
  • D.3 QUESTION FORMATS: MULTIPLE-CHOICE VS. OPEN-ENDED: D.3 QUESTION FORMATS: MULTIPLE-CHOICE VS. OPEN-ENDED — In localization, all five models select the correct timestamp with options, but none produces the true time open-ended and every rubric score is zero.Two models quote the event’s opening sentence, identifying its content while failing to locate it temporally.
  • D.4 RUBRIC AXES: SEMANTIC VS. TEMPORAL: D.4 RUBRIC AXES: SEMANTIC VS. TEMPORAL — Semantic accuracy is 43–65% and declines gently with length, whereas temporal accuracy starts lower and falls as audio grows.For Gemini 2.5 Pro, performance drops from 42% at S to 23% at L, while wrong timing rises from 46% to 67%.
  • D.4 RUBRIC AXES: SEMANTIC VS. TEMPORAL: D.4 RUBRIC AXES: SEMANTIC VS. TEMPORAL — Essential criteria score 50.7 versus 42.5 for supporting criteria, despite semantic accuracy of 58.8 versus 52.9.The difference reflects timing load: almost every supporting criterion demands a timestamp.

D.5 CHAIN SCORING: GROUNDING VS. GUESSING

AudioSpan’s chain score primarily rewards temporal grounding rather than isolated link accuracy: ungrounded guesses rarely form correct prefixes, while anchor injection restores performance. Understanding is generally answered more often after locating the anchor, though one model shows the opposite guessing signature.

  • Grounding versus guessing: Gemini 3.1 Pro thinking answered understanding more often after missing the anchor, by −8.6 points, indicating a guessing signature.The pooled comparison favored located anchors by +13.1 points, but this model inverted the pattern.
  • Chain scoring: A blind solver scored 10.4 on the chain despite clearing each link 22–34% of the time, while anchor injection raised the score to 61.4.Adding the ASR transcript barely increased the score to 12.6; first-error truncation prevents ungrounded guesses from forming a correct prefix.
  • Anchor operations: 43.8% perception accuracy for Modification exceeded 38.7% for Addition and 32.8% for Deletion, although reasoning peaked under Deletion at 42.5%.This ordering held across every model and duration tier; silenced spans were hardest to locate, while altered spans remained easiest.

E LIMITATIONS

AudioSpan’s two question paths are complementary but constrained: Native QA depends on caption fidelity, while Anchor QA trades free-form diversity for deterministic ground truth within a fixed chain template.

  • Question-path limitations: Native QA coverage is limited by caption fidelity, especially for the longest audio where omissions are hardest to detect and unrecoverable through human review.Items can only draw on what the caption records.
  • Question-path limitations: Anchor QA provides deterministic ground truth through synthetic anchors but confines questions to a grounding-to-reasoning chain template.This sacrifices the free-form diversity available in Native QA.
Loading 2608.26431v1…