PaperlayerView exact revision on arXiv

Source-linked AI summary

Logbook: Extremely Long-form Audio Event Understanding

Kwanghee Choi, Suwon Shon, Dmitriy Serdyuk, Guitang Lan, Chao-Wei Huang, Mohammad Sadegh Rasooli, Sangeeta Srivastava, Zhaojiang Lin, Saurabh Adya, Ming Sun

arXiv:2610.07338v1eess.AScs.AIcs.CLcs.LGcs.SD

TL;DR

Existing audio benchmarks largely use short, pre-segmented clips, leaving long-form continuous audio less represented. LOGBOOK defines and evaluates gap-free segmentation with labels and descriptions across hour-scale recordings. The task is tractable, but systems over-segment, and their strengths and weaknesses vary with fine-tuning, context length, and reasoning.

  • Problem

    Many audio datasets use short, pre-segmented clips, and even longer-context audio language models have been evaluated on recordings of at most five minutes.

    Source: §1 · p. 1

  • Method

    LOGBOOK benchmarks gap-free segmentation with an event label and description per segment, using recordings from ten minutes to six days and comparing 52 systems.

    Source: §1 · p. 1

  • Results

    The task is tractable, but systems over-segment; fine-tuning partly mitigates this, end-to-end systems can outperform cascades yet decline with longer windows, and reasoning mainly improves instruction following.

    Source: §1 · p. 1, §6 · p. 4, §5.2 · p. 4

  • Takeaways & Limitations

    The authors identify structured decoding, reinforcement learning, longer-context end-to-end models, and assistive or personalized applications as future directions.

    Source: §6 · p. 4

    On out-of-distribution SINS and EgoLife, fine-tuning gains are much smaller, and Gemma 4 degrades on SINS amid sleep and absence classes missing from Ego4D.

    Source: §5.1 · p. 3

Abstract

from arXiv · show

Audio benchmarks are built around short, pre-segmented clips, limiting model design to brief inputs or fixed vocabularies. To close this gap, we introduce Logbook, a benchmark for hour-scale audio understanding, with recordings ranging from ten minutes to six days. Given a continuous audio recording and an event label vocabulary, a system must predict a gap-free segmentation with an event label and a description per segment. We compare 52 systems, end-to-end and cascaded, and ablate fine-tuning, context length, and reasoning budget. We find the task tractable, though the best systems remain below the human reference. Also, over-segmentation is pervasive, and fine-tuning partially mitigates it. Finally, end-to-end are often better than cascaded systems, but degrades with longer context.

1. INTRODUCTION

LOGBOOK frames long-form audio understanding as gap-free segmentation with an event label and description for every segment. The benchmark compares 52 systems and finds the task tractable, while exposing persistent over-segmentation and limits in system performance.

1. INTRODUCTION

  • Existing audio datasets commonly use short, pre-segmented clips, while continuous-recording datasets are often short, activity-specific, video-centered, or speech-focused.Even audio language models with longer context are evaluated on recordings of at most five minutes.Source: §1 · p. 1
  • LOGBOOK defines long-form audio understanding as gap-free segments, each carrying an event label and description, and benchmarks recordings from ten minutes to six days.The authors compare 48 cascaded systems and four end-to-end models, with ablations of fine-tuning, context length, and reasoning budget.Source: §1 · p. 1
  • Gap-free segmentation is tractable from audio alone, but systems often over-segment; fine-tuning partly mitigates this, and end-to-end systems can outperform cascaded systems yet degrade with longer windows.Reasoning yields limited gains, mainly by improving instruction following rather than audio understanding.Source: §1 · p. 1

2. THE LOGBOOK BENCHMARK

The LOGBOOK benchmark formalizes continuous audio understanding as full-coverage segmentation and adapts several datasets into a common evaluation setting. Its benchmark includes defined preprocessing procedures and metrics for segmentation, descriptions, compute, and downstream utility.

2.1. Task definition (Figure 1)

  • Each output segment has one event label and a free-text description; non-overlapping contiguous segments must cover the entire recording, and the fixed vocabulary sets segmentation granularity.Full coverage prevents systems from increasing precision by abstaining.Source: §2.1 · p. 1

2.2. Datasets

  • Ego4D contributes daily-life recordings and two annotation tiers, EgoLife supplies week-long household audio from six participants, and SINS covers six days of one inhabitant.Ego4D provides summary narrations and localized actions; EgoLife provides dense Mandarin captions, while SINS has contiguous labels but no descriptions.Source: §2.2 · pp. 1–2

2.3. Dataset preprocessing

  • Ego4D narrations are cleaned, decomposed into atomic facts, classified, and merged; EgoLife captions are translated and grouped, while SINS fills unannotated intervals with absence.Ego4D splits are assigned by participant hash at 60/20/20, and test recordings are filtered by duration and event count.Source: §2.3 · p. 2

2.4. Evaluation

  • Evaluation uses frame accuracy (fAcc) and description F1 (dF1) as headline metrics, alongside boundary and event-error measures, fact-level precision and recall, compute, and downstream utility.The human reference is a proxy: its labels are LLM-inferred, annotators saw video, and its five-minute grid differs from minute-resolution systems.Source: §2.4 · p. 2

3. SYSTEMS

LOGBOOK compares systems that process audio directly with cascaded systems that first caption audio windows and then pass captions to a text model. The evaluated systems include long-context audio models, audio captioners, and instruction-tuned text models.

3. SYSTEMS

  • End-to-end systems consume audio directly and predict labeled, described segments, whereas cascaded systems pass audio captions to a text LLM to produce the same output format.Source: §3 · p. 2

3.1. End-to-end systems

  • The end-to-end comparison includes Qwen 2.5-Omni, Qwen 3-Omni, Audio Flamingo 3, and Gemini 3.5 Flash, all used as multimodal audio models.Audio Flamingo 3 aligns a pretrained Whisper encoder with a Qwen decoder; Gemini 3.5 Flash accepts audio natively.Source: §3.1 · p. 2

3.2. Cascaded systems

  • Cascaded systems pair audio captioners—including Qwen and CLAP models—with eight instruction-tuned text models spanning open, open-weight, and closed-API options.Source: §3.2 · p. 2

4. RESULTS

E2E Gemini 3.5 is strongest on Ego4D and EgoLife, while cascaded systems lead on SINS; component results vary, and cascaded systems are not cheaper.

4. RESULTS

  • E2E models process ten-minute windows, while captioners process ten-second windows grouped into ten minutes; boundaries are predicted at minute-level resolution.Source: §4 · p. 2

4.1. Comparing different components of cascaded systems

  • AF 3 leads among captioners on Ego4D and SINS but is computationally heavy; MSCLAP comes close at a fraction of the cost.Source: §4.1 · p. 3, Table 2 · p. 3
  • Qwen 3-o beats its captioner-tuned variant throughout the datasets except SINS.Source: §4.1 · p. 3
  • Among text models, closed APIs are often best, while Gemma 4 is generally the strongest non-API model.Source: §4.1 · p. 3, Table 3 · p. 3
  • K2-V2's low evER reflects no parseable output for 64% of windows; subsequent cascades use AF 3 with Gemini 3.5, GPT 5.6, or Gemma 4.Source: §4.1 · p. 3

4.2. Comparing cascaded systems with E2E systems

  • E2E Gemini 3.5 is best on Ego4D and EgoLife across the compared systems, but cascaded systems lead on SINS.Source: §4.2 · p. 3
  • Qwen 3-o is the strongest non-API E2E model, while AF 3 scores zero after failing to emit the required output schema.Source: §4.2 · p. 3
  • 16.93 PFLOPs for Gemma 4 alone exceeds E2E Qwen 3-o's 11.97 PFLOPs; adding AF 3 raises the cascade to 71.41 PFLOPs.Source: §4.2 · p. 3

5. ANALYSIS

The analysis finds that fine-tuning mitigates over-segmentation, while longer context can hurt end-to-end systems and reasoning yields limited gains. These improvements remain bounded by out-of-distribution data and output-format failures.

5. ANALYSIS

  • Both cascaded and end-to-end systems fall short of the human baseline on Ego4D, and even the best systems over-segment.The analysis tests supervised fine-tuning, longer context, and reasoning as possible improvements.Source: §5 · p. 3

5.1. Supervised fine-tuning (SFT)

  • SFT brings event error rate below 100% for cascaded Gemma 4 and E2E Qwen 3-o on every dataset, while sharply improving Ego4D frame accuracy.The predicted segmentations also become coarser, with fewer events and longer mean event lengths on every dataset.Source: §5.1 · p. 3, Table 4 · p. 4
  • SFT gains are much smaller on out-of-distribution SINS and EgoLife, and Gemma 4 degrades on SINS.The authors link this in part to class coverage: Ego4D lacks sleeping and absence events found in SINS.Source: §5.1 · p. 3

5.2. Context window size

  • Longer windows leave cascaded segmentation similar or better, but end-to-end performance peaks at 5 or 10 minutes and then declines.Description precision tends to rise as recall falls when predicted segments become longer.Source: §5.2 · p. 4

5.3. Reasoning

  • Only E2E Gemini benefits from larger reasoning budgets; other systems are similar or worse, with gains attributed mainly to improved instruction following.Output truncation at the highest budget explains drops for cascaded Gemini 3.5 and GPT 5.6.Source: §5.3 · p. 4, Figure 4 · p. 4

6. CONCLUSION

The conclusion introduces LOGBOOK’s gap-free segmentation task for long-form audio and reports that current models can perform it, but remain prone to over-segmentation. It also summarizes the trade-offs observed across system designs, context lengths, and reasoning, and identifies directions for future work.

  • LOGBOOK defines long-form audio understanding as gap-free segments, each carrying an event label and description.Source: §6 · p. 4
  • The task is feasible with current models, but all systems over-segment, which SFT mitigates.Source: §6 · p. 4
  • End-to-end systems can outperform cascaded systems on headline metrics and efficiency, yet degrade with longer windows; reasoning gains are limited and mainly reflect instruction following.Source: §6 · p. 4
  • Future work could explore additional event vocabularies, structured decoding, reinforcement learning, longer end-to-end context, and assistive applications.Source: §6 · p. 4

7. COMPLIANCE WITH ETHICAL STANDARDS

The study used publicly available datasets and models; the authors state that ethical approval was therefore not required.

  • The study relied entirely on publicly available datasets and models, so ethical approval was not required.Source: §7 · p. 5

A.1. Ablation on context window size for description quality

Longer input windows yield longer predicted events and fewer facts, with description precision generally rising as recall falls. The recall decline dominates, reducing overall description F1.

  • Longer input windows produce longer predicted events across every system and correspondingly fewer facts.Source: §A.1 · p. 6
  • Description precision generally rises while recall falls as input windows grow.Source: §A.1 · p. 6
  • The recall decline dominates the precision increase, so description F1 decreases overall.Source: §A.1 · p. 6

A.2. Ablation on reasoning budget for description quality

Description quality changes with reasoning budget in a pattern similar to segmentation, while parsing failures offer an explanation for the observed changes.

A.2. Ablation on reasoning budget for description quality

  • E2E Gemini 3.5 improves description quality, while cascaded Gemini 3.5 and GPT 5.6 collapse at the largest reasoning budget and other systems remain flat.Source: §A.2 · p. 6
  • Description quality follows nearly the same reasoning-budget trend as segmentation.Source: §A.2 · p. 6
  • Parsing failures, rather than improved understanding, are the authors’ explanation: a window that fails to parse yields neither segments nor facts.Source: §A.2 · p. 6

A.3. Fact statistics

The appendix reports how fine-tuning changes fact and event counts and how reasoning budgets relate to parsing failures. It also documents prompts, cascaded-system coverage, and class-wise accuracy.

A.3. Fact statistics

  • Ground-truth facts average 3.64 per five minutes on Ego4D versus 48.67 on EgoLife; systems produce similar counts, and SFT roughly halves them.Source: §A.3 · p. 6, Table 6 · p. 7

A.4. Event Statistics

  • SFT cuts Ego4D event counts 4.7× for the cascade and 2.1× for E2E; both settle near 0.3 events per five minutes, below the reference.Source: §A.4 · p. 6, Table 7 · p. 7

A.5. Parsing Failure Statistics

  • E2E Gemini’s schema-failure rate falls from 15.4% to 0.6% on Ego4D with reasoning, while largest-budget cascaded Gemini and GPT 5.6 failures rise to 49.2% and 43.1% from truncation.Source: §A.5 · p. 6, Table 8 · p. 7

A.6. Prompts and Class definitions

A.7. Full results of cascaded systems

  • Table 9 covers all 48 cascaded combinations—six captioners paired with eight text models—on Ego4D, SINS, and EgoLife; Tables 2–3 average the grid by axis.Source: §A.7 · p. 9, Table 9 · p. 10

A.8. Class-wise frame accuracy of top-line systems

  • Class-wise fAcc is the fraction of frames of a class labeled correctly; Table 10 compares top-line systems by event class, ordered by frame share, with cascades using Audio Flamingo 3.Source: §A.8 · p. 9, Table 10 · p. 11
Loading 2610.07338v1…