PaperlayerView exact revision on arXiv

Source-linked AI summary

Listen, Reason, and Segment: Aligning LALMs with Editorial Judgment for Media Chapterization

Tony Alex, Wish Suharitdamrong, Sara Atito, Armin Mustafa, Muhammad Awais, Philip J. B. Jackson, Jiankang Deng, Ismail Elezi

arXiv:2608.16539v1cs.SDcs.AIcs.CLeess.AS

TL;DR

Practical media workflows need audio chapterization that reflects subjective editorial judgment rather than only benchmark performance. AudioChaps aligns end-to-end LALMs with creator annotations using reasoning-guided GRPO, improving macro-F1 over its backbone by 49.2 points.

  • Problem

    Audio LALMs show benchmark progress, but practical deployment remains limited because chapter boundaries require subjective editorial judgment over distributed acoustic evidence.

  • Method

    AudioChaps aligns end-to-end LALMs for chapterization by combining structured reasoning supervision with GRPO optimized against creator-authored chapter annotations.

  • Results

    49.2 points: AudioChaps-R1 improves macro-F1 over the AF3-Think-8B backbone, with gains across accuracy, precision, recall, and F1.

  • Takeaways & Limitations

    Chapterization can provide navigable structure for higher-level media applications, while aligned performance remains effective across diverse acoustic regimes.

  • Takeaways & Limitations

    The framework does not natively support long-context audio modelling, limiting globally informed chapter titles and segment summaries.

Abstract

from arXiv · show

Large Audio Language Models (LALMs) have made rapid progress on standardized benchmarks, yet their deployment in practical media workflows, curation, archival indexing, and content distribution remains largely unrealized. We identify automated audio chapterization, the task of segmenting continuous audio streams into thematically coherent chapters, as a demanding and commercially consequential setting that exposes this gap. Chapterization is challenging because boundaries are defined less by objective acoustic events than by subjective editorial judgment, requiring models to reason sequentially over long acoustic contexts and approximate creator-authored boundary decisions. We present AudioChaps, a post-training framework for aligning end-to-end LALMs for this task via Group Relative Policy Optimization (GRPO) guided by Chain-of-Thought (CoT) reasoning. To support training and evaluation, we curate three datasets: AudioChaps-Alignment, derived from creator-annotated chapter boundaries on YouTube; AudioChaps-CoT, which provides structured supervision for well-formatted, high-quality, and evidence-grounded boundary reasoning; and AudioChaps-Eval, a held-out benchmark for audio chapterization. Applying GRPO directly without a Supervised Fine-Tuning (SFT) cold start, AudioChaps-R1-Zero already improves average F1 by 33 points over the state-of-the-art LALM Audio-Flamingo-3-Think. The AudioChaps framework produces our final aligned LALM, AudioChaps-R1, which improves average F1 by 49 points. These results demonstrate that GRPO-trained LALMs can reliably transform unstructured auditory streams into navigable, structured media. Our code, models, and dataset resources will be released upon acceptance at https://github.com/ta012/AudioChaps.

1 Introduction

The section argues that audio chapterization is a commercially important but difficult test of real-world LALM utility because subjective editorial boundaries require long-context reasoning across heterogeneous audio. It introduces AudioChaps, its datasets, and an aligned model that achieves state-of-the-art chapterization performance.

  • Motivation: Audio chapterization partitions arbitrary-duration audio into thematically coherent chapters reflecting latent structure that human editors would impose.The task is broadly applicable but requires reasoning over continuous acoustic streams rather than relying only on objective acoustic events.
  • Motivation: End-to-end LALMs are targeted because ASR-plus-text-LLM pipelines degrade sharply on heterogeneous media such as dynamic editorial content, music, and gaming streams.The proposed system operates directly on raw audio and is intended to generalize across these regimes.
  • Method: AudioChaps uses GRPO to align LALMs with creator-authored editorial annotations, producing the aligned AudioChaps-R1 model.The method optimizes final boundary decisions because chapter boundaries reflect subjective editorial judgment and distributed acoustic evidence.
  • Datasets: The framework introduces AudioChaps-Alignment, AudioChaps-CoT, and AudioChaps-Eval for creator-annotated training, evidence-grounded boundary reasoning, and audio-only evaluation.AudioChaps-Alignment spans structured speech, dynamic media, gaming, and music, while AudioChaps-Eval addresses the lack of a standardized benchmark.
  • Results: 28.6 to 77.8 average F1 over its base model establishes state-of-the-art performance across all four acoustic regimes for AudioChaps-R1.AudioChaps-R1 also surpasses a 32B RL-trained LALM at roughly a quarter of the parameters.

2 Related Works

Prior chapterization research has focused almost entirely on video, progressing from large user-chaptered YouTube corpora to systems handling hour-long content with multimodal supervision. Existing deployed tools remain transcript-driven.

  • Audio and Video Chapterization: Chapterization has been studied almost entirely in the video domain.
  • Audio and Video Chapterization: VidChapters-7M introduced a large corpus of user-chaptered YouTube videos and tasks for chapter generation, boundary-conditioned generation, and grounding.
  • Audio and Video Chapterization: More recent systems extend to hour-long content by combining ASR transcripts and visual tokens under instruction tuning.

3 Methodology

AudioChaps formulates chapterization as binary boundary detection over 60-second audio clips and aligns AF3-Think-8B through evidence-grounded CoT supervision followed by GRPO calibration. The method adapts rewards to the backbone’s native outputs while using creator-authored chapter markers as editorial supervision.

  • Task formulation: Audio chapterization is framed as boundary detection over 60-second clips, with positive boundaries sampled between 20 and 40 seconds and negatives buffered within chapters.Positive windows provide at least 20 seconds of acoustic context on both sides of the transition.
  • Task formulation: Creator-annotated YouTube chapter markers provide the ground-truth proxy for human editorial judgment, while training optimizes a flow-based boundary-presence verdict rather than timestamps.The timestamp-free objective addresses AF3-Think-8B’s insufficient temporal accuracy for reliable reasoning supervision.
  • Backbone and baseline: AF3-Think-8B is the primary backbone, and its zero-shot chapterization performance establishes the unaligned reference point for measuring AudioChaps improvements.Step-Audio-R1-32B is additionally evaluated and supplies acoustic perception logs during AudioChaps-CoT curation.
  • Direct GRPO alignment: Direct GRPO rewards a discernible reasoning trace and correct binary verdict instead of enforcing the backbone’s incompatible strict think-answer schema.This preserves rule-based reinforcement learning while avoiding collapse of the learning signal from penalized initial rollouts.
  • Two-stage alignment: The primary AudioChaps-R1 procedure fine-tunes AF3-Think-8B on AudioChaps-CoT for structured reasoning, then applies GRPO to calibrate boundary decisions against creator-authored annotations.SFT standardizes reasoning and verdict separation, while GRPO addresses imitation without sufficient decision calibration.
  • CoT data construction: AudioChaps-CoT uses an audio-to-text bridge that generates pseudo-CoT, sanitizes it into chronological acoustic perception logs, and synthesizes structured evidence-grounded targets.The targets use flow-based language rather than exact timestamps and support the SFT cold start.

4 Experiments

Experiments evaluate AudioChaps across heterogeneous acoustic subtypes using creator-authored, source-disjoint clip data and balanced full-length recordings. AudioChaps-R1 substantially outperforms zero-shot baselines, improves long-form chapterization, and benefits from both SFT initialization and GRPO calibration.

  • Evaluation Data: AudioChaps-Eval contains approximately 16k held-out clips from 749 source videos, with 7,011 positive and 8,952 negative clips split at the source-video level.The full-length evaluation additionally uses 40 recordings, 10 per acoustic subtype, containing 387 reference boundaries and approximately 33 hours of audio.
  • Zero-Shot Baselines: Zero-shot AF3-Think-8B is highly conservative, achieving 64.0 precision but 3.2 recall on Music, while Step-Audio-R1-32B remains between 60% and 65% accuracy across subtypes.Recall also remains below 25 in Structured Speech and Dynamic Media for AF3-Think-8B.
  • Clip-Level Results: AudioChaps-R1 raises average accuracy from 53.2 to 78.8, precision from 51.0 to 74.9, recall from 25.2 to 81.2, and F1 from 28.6 to 77.8 over AF3-Think-8B.A paired video-level bootstrap analysis reports a 49.2-point macro-F1 improvement with 95% confidence interval [46.2.
  • Clip-Level Results: AudioChaps-R1 narrows cross-subtype F1 variation to 11.2 points, ranging from 73.4 on Dynamic Media to 84.6 on Music.AF3-Think-8B spans 43.9 F1 points, from 6.0 on Music to 49.9 on Gaming.
  • Full-Length Evaluation: AudioChaps-R1-8B reaches 37.6 full-length F1 versus 6.5 for AF3-Think-8B and 9.5 for fixed 180-second intervals, reducing median deviation from 38.0 to 10.0 seconds.Inference uses sliding 60-second windows with a 20-second hop, grouping consecutive positive windows into one boundary estimate.
  • Training Ablations: GRPO after the CoT cold start provides the best trade-off: compared with AudioChaps-SFT-8B, precision rises by 12.1, 11.2, 20.7, and 2.5 points across Dynamic Media, Gaming, Music, and Structured Speech.Direct GRPO without SFT reaches F1 scores of 57.8, 64.4, 59.1, and 62.9 across those subtypes, while SFT reaches 70.0, 72.5, 74.1, and 77.9.

5 Conclusion … A.2 Hyperparameters and Training Budget

The paper positions audio chapterization as a practical media workflow requiring alignment with creator-authored editorial judgment. It combines structured reasoning supervision with GRPO, while noting limitations in long-context audio modeling and detailing its training prompts and compute budget.

  • 5 Conclusion: Audio chapterization is studied as a setting for moving LALMs beyond benchmark performance toward practical media workflows.The task treats chapter-boundary placement as editorial judgment rather than a fixed acoustic event.
  • 5 Conclusion: The framework aligns end-to-end LALMs with creator-authored annotations using AudioChaps-Alignment, AudioChaps-CoT, and AudioChaps-Eval.Its two-stage recipe combines a CoT SFT cold start with GRPO alignment.
  • 5 Conclusion: The framework does not natively support long-context audio modeling, limiting globally informed chapter titles and segment summaries.Native long-context modeling and full chapter generation are identified as directions toward production deployment.
  • A.1 Training Prompt Template: Both training stages use a unified prompt framing chapterization as a single-choice question with deliberate, human-like internal reflection.The prompt includes natural cognitive markers such as “let me think,” “wait,” and “let’s break it down.”
  • A.1 Training Prompt Template: Prompts require self-verification and restrict outputs to <think>...</think> reasoning traces followed by a single option letter in <answer>...</answer>.The full prompt is shown in Figure 3.
  • A.2 Hyperparameters and Training Budget: SFT used 8 NVIDIA H200-140GB GPUs for approximately 4 hours, while GRPO used 8 nodes with 4 NVIDIA GH200-96GB each for approximately 10 hours.Both stages fine-tune the 8B-parameter backbone rather than training from scratch, keeping the overall budget modest relative to comparable pretraining.

B Detailed Ablation Results · Training prompt template

This block details the per-subtype, per-metric evaluation underlying the main-paper F1 summary and describes the unified prompt used for SFT and GRPO. It covers accuracy, precision, recall, and F1 across pipeline variants and content subcategories.

  • B Detailed Ablation Results: Table 5 breaks down accuracy, precision, recall, and F1 for AF3-Think-8B and three pipeline variants.The variants are R1-Zero, SFT, and the full R1 model.
  • B Detailed Ablation Results: The detailed results provide the per-subtype, per-metric breakdown behind the main-paper F1 summary.
  • Training prompt template: The unified prompt template is used in both the SFT cold-start and GRPO stages.
  • Training prompt template: The model answers a binary single-choice question, reasons with reflective markers inside <think> tags, and emits one option letter inside <answer> tags.
  • B Detailed Ablation Results: Precision and recall behavior can be read directly across the content subcategories.The breakdown is intended to support the precision and recall analysis discussed in Section 4.3.
  • B Detailed Ablation Results: The evaluation organizes results across content subcategories.

C Backbone Generalization of AudioChaps

AudioChaps is designed as a GRPO-based framework that adapts different LALM backbones to creator-authored editorial annotations for audio chapterization. Across AF3-Think-8B and MOSS-Think-8B, GRPO calibrates recall-heavy predictions toward more selective and balanced boundary detection.

  • Backbone adaptation: AudioChaps adapts AF3-Think-8B and MOSS-Think-8B using backbone-specific initialization before GRPO.AF3-Think-8B requires a CoT SFT cold start, whereas MOSS-Think-8B’s initialization is adapted to its existing capabilities.
  • Precision-recall calibration: 88.2 macro-average recall and 63.3 precision characterize the AF3-based SFT model, while GRPO raises precision to 74.9 and F1 from 73.6 to 77.8, reducing recall to 81.2.This reflects calibration from a strongly recall-oriented policy toward more selective predictions.
  • Cross-backbone calibration: GRPO corrects an overly recall-heavy decision policy and produces a more selective and balanced chapter-boundary detector across distinct LALM families.The same precision-recall behavior is reported for MOSS-Think-8B as for the AF3-based model.
  • MOSS generalization: 9.9 points on Dynamic Media, 5.3 on Gaming, 9.9 on Music, and 6.3 on Structured Speech are the MOSS-based model’s F1 gains across all four subcategories.The gains support a similar GRPO calibration role across distinct LALM families.

D Comparison with Larger General-Purpose Multimodal and Omni-Modal Models … F.3 Matching and Metrics

The paper evaluates AudioChaps against larger general-purpose models, tests the statistical reliability of its gains, and extends chapterization from clips to full-length recordings. It defines recording-level inference, decoding, matching, and macro-averaged metrics for this setting.

  • D Comparison with Larger General-Purpose Multimodal and Omni-Modal Models: AudioChaps-R1 with the MOSS backbone achieves the best macro-average accuracy, precision, and F1 against Gemini 2.5 Flash and Qwen3-Omni-30B-A3B-Thinking.Despite using 8B parameters, AudioChaps-R1 reaches 83.4 accuracy, 82.4 precision, and 81.4 F1; it improves macro-average F1 over Gemini 2.5 Flash by 4.6 points.
  • D Comparison with Larger General-Purpose Multimodal and Omni-Modal Models: Task-specific alignment enables a compact open model to outperform substantially larger general-purpose LALMs on audio chapterization.The comparison measures specialized alignment rather than the ultimate capability of stronger or larger backbones, which may improve further when adapted with AudioChaps.
  • E Statistical Significance of the Main Improvement: A paired non-parametric bootstrap compares AF3-Think-8B and AudioChaps-R1-8B while preserving dependencies among clips from the same source video.The analysis resamples source videos rather than individual clips and uses 10,000 paired iterations, computing positive-class F1 for each acoustic subtype before macro-averaging.
  • E Statistical Significance of the Main Improvement: 49.2 F1 points separate AudioChaps-R1-8B from AF3-Think-8B, with a 95% bootstrap confidence interval of [46.2, 52.2] and a two-sided p-value below 10^-4.AudioChaps-R1-8B obtains macro-F1 77.8 versus 28.6 for AF3-Think-8B; subtype improvements are also significant, including +78.6 for Music.
  • F Full-Length Audio Chapterization Evaluation: The full-length evaluation tests whether local chapter-boundary decisions can be converted into recording-level boundaries on uncropped audio.This setting is harder because boundaries are sparse, recordings may contain multiple boundaries, and overlapping window predictions must be consolidated into a sequence.
  • F.1 Evaluation Set and Sliding-Window Inference: The evaluation uses 40 full-length recordings totaling approximately 33 hours, with 387 reference chapter boundaries across four acoustic subcategories.There are 10 recordings per subcategory, and the mean recording duration is 49 minutes 29 seconds; inference uses overlapping 60-second windows with a 20-second hop, producing 5,875 windows.
  • F.2 Recording-Level Decoding: Binary boundary predictions are decoded by grouping consecutive positive windows into runs, placing a boundary at each run’s midpoint, and merging estimates less than 20 seconds apart by their median.The window-run decoder prevents a single transition detected across overlapping windows from being counted multiple times.
  • F.3 Matching and Metrics: Predicted and reference boundaries are greedily matched one-to-one within a ±10-second tolerance, after ordering candidate matches by absolute temporal distance.Precision, recall, and F1 are computed per recording, macro-averaged within each subtype, and averaged equally across subtypes; dev-R2E, prediction-to-reference count ratio, and a one-boundary-per-180-seconds baseline are also reported.

F.4 Results

AudioChaps substantially improves full-length audio chapterization across evaluated backbones, increasing F1 and reducing boundary-timing error. These gains show that clip-level or local boundary training transfers to recording-level predictions.

  • Full-length chapterization: 37.6 average F1: AudioChaps-R1-8B surpasses the fixed-interval baseline’s 9.5 and AF3-Think-8B’s 6.5, which predicts no boundaries on 15 of 40 recordings.It also reduces pooled median reference-to-estimate deviation from 38 to 10 seconds.
  • Full-length chapterization: 46.2 F1: MOSS-Think-AudioChaps-R1-8B improves over MOSS-Think-8B’s 26.4, primarily through a precision increase from 22.4 to 48.2.The result indicates that locally trained boundary judgments can be consolidated into effective recording-level predictions.

G Training and Evaluation Dataset Statistics

The dataset is curated from VidChapters-7M and stratified into four acoustic subtypes. Training and held-out test splits are analyzed by subtype, subcategory, and positive/negative clip balance.

  • The data is curated from VidChapters-7M and stratified into four acoustic subtypes.
  • The dataset breakdown covers subcategories and positive-to-negative clip balance.
  • Figure 4 reports per-subtype clip counts for training and held-out test splits.

H Sample Output Comparison: AF3-Think vs. AudioChaps-R1 with Human Evaluation · I Additional Temporal-Localization Analysis · I.1 MOSS Timestamp Supervision

Blind human evaluation favored AudioChaps-R1 over AF3-Think for chapter-boundary judgments and justification quality. For MOSS-Think-8B, auxiliary timestamp supervision substantially improved temporal localization across all four subcategories.

  • H Sample Output Comparison: AF3-Think vs. AudioChaps-R1 with Human Evaluation: Seven PhD student or post-doctoral raters evaluated anonymized AF3-Think and AudioChaps-R1 responses using 60-second audio clips and chapter-boundary questions.Responses were randomized to remove position bias.
  • H Sample Output Comparison: AF3-Think vs. AudioChaps-R1 with Human Evaluation: On a 1–5 scale, AF3-Think scored 2.77 while AudioChaps-R1 scored 4.46 for boundary correctness and justification quality.The averaged scores covered all samples and raters.
  • I Additional Temporal-Localization Analysis: AudioChaps primarily determines whether a chapter boundary occurs within an approximately ±10-second interval for practical chapter-level navigation.More precise temporal localization is described as potentially valuable but not central to the primary task.
  • I.1 MOSS Timestamp Supervision: MOSS-Think-8B provides native temporal localization by emitting an estimated boundary location within each 60-second input window.AudioChaps exploits this capability alongside alignment to creator-authored chapter-boundary decisions.
  • I.1 MOSS Timestamp Supervision: For MOSS, AudioChaps adds a small auxiliary timestamp reward based on the creator-annotated boundary timestamp alongside the boundary-classification objective.This supervision is designed to use MOSS-Think-8B’s temporal-localization capability.
  • I.1 MOSS Timestamp Supervision: MAE decreased from 7.39 to 2.88 seconds on Dynamic Media, 9.42 to 4.49 seconds on Gaming, 5.47 to 2.46 seconds on Music, and 6.75 to 2.98 seconds on Structured Speech.The AudioChaps-adapted MOSS model improved temporal localization across all four subcategories.
  • I.1 MOSS Timestamp Supervision: These results show that GRPO-based AudioChaps leverages MOSS-Think-8B’s native temporal-localization capability for more accurate chapter-boundary localization.The improvement is reported across all four evaluated subcategories.

I.2 Timestamp-Aware Recording-Level Decoding

Timestamp-aware decoding converts in-window timestamps into absolute boundary votes and improves recording-level chapter-boundary detection for MOSS-Think-AudioChaps-R1-8B. It raises precision, recall, and F1 while reducing median boundary deviation, with additional gains when temporal outputs are reliable.

  • Decoding procedure: For each positive window prediction, MOSS converts its in-window timestamp into an absolute boundary vote at t0 + ts, then clusters votes using a 10-second linkage threshold.This timestamp-based decoder is evaluated separately from the primary window-run decoder.
  • Decoding results: F1 rises from 46.2 to 55.2 for MOSS-Think-AudioChaps-R1-8B under timestamp-aware decoding.Precision increases from 48.2 to 54.3, while recall increases from 49.3 to 64.9.
  • Decoding results: 2 seconds is the pooled median reference-to-estimate deviation, reduced from 8 seconds with timestamp-aware decoding.Among clusters supported by multiple overlapping windows, the median within-cluster spread is 1 second and the 90th-percentile spread is 8 seconds.
  • Interpretation: Timestamp-aware decoding provides additional full-length gains when the underlying LALM supports reliable temporal outputs.The results complement the primary window-run evaluation without changing AudioChaps’s central focus on creator-authored editorial annotations.
Loading 2608.16539v1…