AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMs

1 INTRODUCTION

• Existing benchmarks predominantly use short clips and do not rigorously assess long-span coherence, reasoning, or computational efficiency.Long-form audio also raises computational and memory costs and challenges models to capture extended temporal dependencies.Source: §1 · pp. 1–2
• AudioMarathon evaluates LALMs with 90.0–300.0-second audio, speech, environmental sounds and music, and tasks requiring multi-hop inference.Its tasks span Speech Context Understanding, Audio Scene Understanding, and Voice Characteristic Identification.Source: §1 · p. 2
• AudioMarathon finds performance degradation as audio input length increases and substantial performance gaps among current models in long-context scenarios.The paper also analyzes token pruning and KV-cache eviction trade-offs.Source: §1 · p. 2
• The authors release the benchmark to the community and aim to foster scalable, efficient, and robust LALMs for complex audio understanding.The introduction presents the benchmark as a unified evaluation suite intended to catalyze future research.Source: §1 · p. 2

2 AUDIOMARATHON

• Figure 3 presents per-dataset duration and average audio length in AudioMarathon.Source: Figure 3 · p. 3

2.1 OVERVIEW

• AudioMarathon addresses the short-clip focus of existing benchmarks with a long-form evaluation suite; Figure 2 summarizes its six-stage construction pipeline.The paper also points to Tables 1 and 2 for comparisons with existing benchmarks.Source: §2.1 · p. 3

2.2 DATA COLLECTION AND ANNOTATION

• The six-stage construction process selects ten dataset subsets, concatenates clips, creates task-specific prompts and options, manually verifies samples, and finalizes 6,567 instances balanced across 10 tasks and audio types.At least 20 samples or 10% per sub-dataset were reviewed; finalized audio durations range from 90.0 to 300.0 seconds.Source: §2.2 · pp. 3–4

2.3 COMPARISON WITH OTHER BENCHMARKS

• AudioMarathon targets 90.0–300.0-second recordings, compared with the second-level clips common in public benchmarks; BLAB is long but speech-centric.The benchmark supports flexible duration control.Source: §2.3 · p. 4
• AudioMarathon covers speech, sound, and music with balanced sampling, whereas most benchmarks cover only one or two domains.Source: §2.3 · p. 4
• Its audio version of RACE, generated with text-to-speech, preserves multi-hop reasoning while adding long-term acoustic dependencies.The authors describe it as a stricter test of comprehension, memory, and reasoning.Source: §2.3 · p. 4

• The benchmark comparison includes 16 recent LALMs: ten open-source and six closed-source models.Source: §3 · p. 4
• The listed model examples include Phi-4-Multimodal, Qwen2.5-Omni-3B, Aero-1-Audio, Gemini models, and GPT-4o.Aero-1-Audio is described as compact and audio-centered; Gemini-2.5-Flash and Gemini-2.0-Flash are optimized for faster inference.Source: §3 · p. 5
• Task performance uses F1-score for classification and MCQs, Word Accuracy Rate for ASR, and macro F1-score for audio event detection; efficiency uses latency, peak GPU memory, and speedup.Source: §3 · p. 5
• ASR is evaluated on a filtered LibriSpeech-long test subset, while other tasks use multiple-choice questions with randomized option order.The number of options varies by task: two for SD and SGR, five for SAR, and four for the remaining tasks.Source: §3 · p. 5

4 EFFICIENCY OPTIMIZATION FOR LALMS

• The efficiency study evaluates prefilling-stage token pruning and decoding-stage KV-cache eviction to address the costs of extended audio sequences.A five-minute input can generate thousands of tokens, with quadratic memory growth and prohibitive latency.Source: §4 · p. 5

4.1 TOKEN PRUNING AND KV CACHE EVICTION

• Four token-pruning methods and four KV-cache eviction strategies are tested on three open-source LALMs.Source: §4.1 · p. 6
• Frame consistently outperforms other token-pruning methods across latency constraints on four representative datasets.Source: Figure 5 · p. 6
• Frame uses time-aligned pruning to preserve rare or short-lived acoustic events that random, attention-based, or redundancy-guided pruning may discard.The paper motivates this design by the sequential nature of acoustic signals and their temporal continuity.Source: §4.1 · pp. 6–7
• The figures examine pruning acceleration across four datasets and peak GPU-memory use under eviction during prefilling.The acceleration figure compares pruning strategies; the memory figure illustrates compression benefits without reporting numerical values in the supplied passages.Source: Figure 6 · p. 7, Figure 7 · p. 7
• For KV-cache eviction, the evaluated policies include random removal, key-vector-norm ranking, latest-query attention, and SnapKV’s cumulative-attention scoring with local-neighbor retention.Source: §4.1 · p. 7

• 70.5 average F1-score for Qwen2.5-Omni-7B versus 87.6 for human evaluation; models still struggle with long-form speech understanding.The table compares model performance across speech extraction, audio classification, and speaker information tasks.Source: §5 · p. 7
• All models score below 65 F1 on speaker-related tasks, while the strongest model reaches 83.5 and the top four LALMs exceed 70 on audio classification.The paper reports closed-source models at around 60 F1 on semantically sensitive tasks and identifies speaker information modeling as under-emphasized.Source: §5 · p. 8, Figure 8 · p. 8
• Qwen2.5-Omni-3B’s Speech Understanding F1-score rises from 53.8 to 58.6 with pruning, while Multi-Domain Classification improves from 79.8 to 81.9 and Authenticity Recognition from 67.3 to 72.0.Source: §5 · pp. 8–9, Figure 9 · p. 8
• Pruning sensitivity varies by task and model: fine-grained speech tasks degrade when unique phonetic cues are removed, while global classification can remain resilient under aggressive compression.Frame has the strongest speech-sensitive F1 scores across models and pruning ratios; Aero-1-Audio and Phi-4-Multimodal show speech-understanding declines under stronger or DART-based pruning.Source: §5 · p. 9

• Audio-language modeling progressed from ASR/TTS combinations with text LLMs to self-supervised speech representations and instruction-tuned models that unify audio and language (Baevski et al., 2020; Hsu et al., 2021).Source: §6 · p. 9
• Audio benchmarks expanded from task-specific datasets to broader speech, question-answering, and multimodal instruction-following evaluations, but few directly test long-audio comprehension or efficiency.Examples include AudioSet, SUPERB (Yang et al., 2021), MMAU (Sakshi et al., 2024), and AIR Bench (Yang et al., 2024).Source: §6 · p. 9
• Token compression research includes KV-cache eviction and token pruning, but audio-token compression remains limited and transfer from vision or text tokens is uncertain.The passage describes SnapKV (Li et al., 2024b), H2O, and StreamingLLM (Xiao et al., 2023) as KV-cache approaches.Source: §6 · p. 9

7 CONCLUSION

• Existing LALMs struggle with long-span speech understanding and speaker analysis, showing limitations in consistency and entity tracking and a significant gap from human performance.The conclusion presents these findings as potential directions for future research.Source: §7 · p. 10
• AudioMarathon covers minute-scale speech, sound, and music inputs across ten representative audio tasks.Source: §7 · p. 10
• The conclusion identifies substantial redundancy in audio tokens and presents AudioMarathon as a foundation for robust, efficient long-audio modeling.Source: §7 · p. 10

ETHICS STATEMENT

• The authors identify deepfake audio generation and privacy erosion as potential dual-use risks.Source: ETHICS STATEMENT · p. 10
• They justify public disclosure as a way to foster robustness and safety through transparent benchmarking and highlight limitations for proactive risk mitigation.Source: ETHICS STATEMENT · p. 10
• The authors encourage responsible future work across diverse languages and contexts and state that the research followed academic ethics.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• Experimental hyperparameters, model descriptions, pruning strategies, and evaluation protocols are specified in the main text, with datasets and construction details in Appendix D.Appendices also contain model details, random-baseline results, error analysis, encoding granularity, and pruning analyses.Source: REPRODUCIBILITY STATEMENT · p. 10

A ADDITIONAL RESULTS

• Table 5 provides core statistics for AudioMarathon.Source: Table 5 · p. 16
• Early-stage audio-token pruning is described as removing ineffective tokens to accelerate subsequent layers while maintaining robust performance.Source: A ADDITIONAL RESULTS · p. 17

A.1 RANDOM CHOICE BASELINE

• The random-baseline results report lower F1-scores, which the authors say reflects F1-score's balance of precision and recall.Table 6 covers nine subsets after 100 random selections.Source: §A.1 · p. 17, Table 6 · p. 17

A.2 RESULTS OF QWEN2.5-OMNI-3B ON THE OTHER SIX DATASETS IN AUDIOMARATHON

• Figure 10 compares latency-performance trade-offs on SED, SD, and SAR, while Figure 11 covers MC, ER, and SER.Source: Figure 10 · p. 17, Figure 11 · p. 18
• Frame performs stably and surpasses random methods on SED and MC, but underperforms on the more challenging SER and SAR tasks.The passage also characterizes Frame as relatively reliable overall.Source: §A.2 · p. 17
• Different token-pruning methods show consistent acceleration performance and reduce inference time for long-audio MCQ tasks.Figure 12 covers acceleration effects on SED, SD, and SAR; Figure 13 is also cited in the result passage.Source: §A.2 · p. 17, Figure 12 · p. 18
• Table 7 summarizes error analysis for Phi-4-multimodal on ASR tasks.Source: Table 7 · p. 18

B ERROR ANALYSIS

• Attention-based selection probably removes high-frequency words, and ASR performs worse than other methods at the same pruning ratio.The error analysis compares token-pruning results on ASR.Source: B ERROR ANALYSIS · p. 18

C ENCODING GRANULARITY OF LALMS

• Except for Phi-4-Multimodal, models produce about 7,500 audio tokens for a five-minute clip, including for simple gender or age classification.The authors interpret this as substantial redundancy in current audio encoding; Table 8 reports maximum audio length, maximum encoded tokens, and token rate.Source: C ENCODING GRANULARITY OF LALMS · p. 19, Table 8 · p. 19

D DATASET CONSTITUTE

• Dataset examples cover sentiment, speaker age and gender, spoken RACE reasoning, concatenated LibriSpeech ASR, domestic sound events, and urban acoustic scenes.The sources include SLUE, Vox_Age and Vox_Gender, RACE, LibriSpeech-long, DESED, and TAU Urban Acoustic Scenes.Source: D DATASET CONSTITUTE · pp. 19–20

E MODEL DETAILS

• Model profiles describe varied multimodal designs, including Qwen2.5-Omni’s TMRoPE and Thinker–Talker architecture, Audio-Flamingo models, and Baichuan-Omni-1.5’s audio tokenizer.Audio-Flamingo-2 is described as supporting 30-second-to-five-minute audio, while Audio-Flamingo-3 supports audio understanding up to 10 minutes.Source: E MODEL DETAILS · pp. 20–21

F PROMPT

• The appendix provides prompt templates for SED, MC, SD, ASR, SCR, SER, ASC, SGR, ER, and SAR.The text introduces templates for the benchmark’s various tasks.Source: F PROMPT · p. 22, Figure 14 · p. 22, Figure 15 · p. 22

G.1 LIMITATIONS

• The benchmark focuses on English, leaves some domains and tasks underrepresented, and omits training-time efficiency and energy use.The authors also note dataset license limits, possible bias from automatic pipelines, and that common metrics do not fully capture human preference or safety risks.Source: §G.1 · p. 26

G.2 FUTURE WORK

• Future work includes broader tasks and languages, more compression methods, human evaluation, and energy and cost reporting; diffusion models are proposed as a possible alternative to autoregressive audio LLMs.The authors observe accuracy and efficiency challenges in long-form audio processing, exemplified by ASR.Source: §G.2 · p. 26

H USE OF LLMS

• LLMs assisted with grammar checking and sentence polishing, while humans constructed and carefully reviewed the main code.Auxiliary AI coding tools were used for debugging, code-error analysis, and implementation assistance.Source: H USE OF LLMS · p. 26