PaperlayerView exact revision on arXiv

Source-linked AI summary

Closing the Verification Loop: Self-Check Captioning for Long-Paragraph Detailed Audio Captioning

Fengji Ma, Yan Rong, Xu Li, Chen Zhang, Pengfei Wan, Li Liu

arXiv:2608.30713v1cs.SD

TL;DR

Long-paragraph detailed audio captioning lacks data combining long clips, paragraph captions, and transcript fidelity, and remains vulnerable to generation-mode failure. SCC addresses these gaps with lifecycle-wide audio-grounded verification, producing LACap-50k, LC-SFT, and SCC-Verifier. Across benchmarks, the system achieves state-of-the-art open-source results, while scaling comparable pretraining data remains prohibitively expensive.

  • Problem

    Long-paragraph detailed audio captioning lacks public data combining long clips, paragraph captions, and transcript-level fidelity, while free-form generation remains weaker than audio perception alone.

  • Method

    Self-Check Captioning applies audio-grounded question answering as verification during data construction, training, and inference through LACap-50k, LC-SFT, and SCC-Verifier.

  • Results

    SCC attains state-of-the-art open-source results on MMAU, MMAR, MMSU, and Omni-Cloze, and improves Gemini3.1-Pro by 8.6 Omni-Cloze points when applied at data and inference stages.

  • Takeaways & Limitations

    LACap-50k fills a resource gap for long-paragraph detailed audio captioning research and supports paragraph-level training and fine-grained evaluation construction.

  • Takeaways & Limitations

    Building a corpus of comparable quality at pretraining scale remains prohibitively expensive, so extending SCC to that scale is left for future work.

Abstract

from arXiv · show

Long-paragraph detailed audio captioning, which requires dense and transcript-faithful descriptions of fine-grained audio content, remains unsolved for current audio-visual multimodal language models. We attribute this failure to two structural problems. The first is data poverty, as no public corpus jointly provides long clips, paragraph captions, and verbatim-transcript fidelity. The second is generation-mode failure, evidenced by a 44.8 to 46.4 percentage-point gap between right-audio and shuffled-audio multiple-choice question (MCQ) accuracy. We address both within Self-Check Captioning (SCC), a unified framework that instantiates audio-grounded question answering as the verification primitive at every lifecycle stage. SCC yields three artifacts. Long-paragraph Audio Caption 50k (LACap-50k) is a 50,222-clip audio-visual corpus with 491.5-word captions and a post-hoc automatic speech recognition (ASR) audit. Layer-Curvature Supervised Fine-Tuning (LC-SFT) is the first on-policy supervised fine-tuning method to weight tokens by intermediate-layer evidence, motivated by our identification of Late-Layer Semantic-Entropy Collapse (SEC). SCC-Verifier arbitrates among caption rollouts via audio-grounded self-answering at inference. Across multiple benchmarks, our system attains state-of-the-art among open-source captioners and is competitive with proprietary baselines. We release LACap-50k to fill the resource gap for long-paragraph detailed audio captioning research.

1 Introduction

Long-paragraph detailed audio captioning is limited by missing transcript-faithful long-form data and a generation-mode bottleneck. Self-Check Captioning addresses both through audio-grounded verification across data construction, training, and inference.

  • Motivation: No public corpus jointly provides long clips, paragraph-length captions, and transcript-level fidelity for training detailed audio captioners.Existing corpora trade off caption length, clip duration, coherence, or transcript verification.
  • Motivation: 44.8–46.4 percentage-point gaps between right-audio and shuffled-audio MCQ accuracy expose generation-mode failure beyond a universal recognition-format effect.The broader setting also shows Qwen2.5-Omni-3B/7B reaching 67.0%/75.0% on MCQ but 13.8%/25.8% in cloze form.
  • Motivation: Late-Layer Semantic-Entropy Collapse identifies hallucinated tokens that remain uncertain through middle and late layers before committing at the final one or two layers.This signature is not captured by reweighting based only on final-layer probability.
  • Self-Check Captioning: Self-Check Captioning uses audio-grounded question answering as a verification primitive throughout caption data construction, training, and inference.Its lifecycle includes self-checking during data creation, MCQ-based reward reweighting during training, and SCC-Verifier at inference.
  • Self-Check Captioning: LACap-50k contains 50,222 audio–visual clips with 491.5-word average captions and post-hoc transcript-level ASR verification.The corpus is released to fill the resource gap and support paragraph-level captioner training and fine-grained evaluation resources.
  • Self-Check Captioning: LC-SFT weights tokens using intermediate-layer evidence, while SCC-Verifier selects among caption rollouts through audio-grounded MCQ answering.Together, these components target the identified reliability problem during training and inference.

2 Related Work

Prior audio captioning resources and models largely emphasize short-form or fragmented descriptions, while existing on-policy methods rely on final-layer signals. LACap-50k and LC-SFT address these gaps with long, audited captions and intermediate-layer reliability evidence.

  • Audio captioning resources: Existing audio corpora range from short single-sentence and LLM-distilled captions to multi-expert or long-clip collections without the full target combination.The related-work landscape includes audio-only, multi-expert, long-clip, and agentic audiobook systems.
  • Audio captioning resources: None of the reviewed corpora pairs paragraph-length detail with predominantly long clips and post-construction transcript-level auditing.LACap-50k is positioned as filling this resource gap.
  • Training and verification: On-policy supervised fine-tuning methods reweight sampled tokens from final-layer probabilities, whereas LC-SFT uses late-layer semantic-entropy collapse as an in-backbone reliability signal.The paper presents this intermediate-layer signal as invisible to final-layer reweighting.

3 Self-Check Captioning (SCC)

SCC builds LACap-50k through self-checking data construction, then trains captioners with verifiable rewards and layer-curvature token weighting before verifying rollout candidates at inference.

  • 3.1 Data Stage: LACap-50k Construction: LACap-50k contains 50,222 audio–visual clips paired with 491.5-word average captions and 78.6k unique sub-word tokens.The corpus is designed for long-paragraph audio captioning and is paired with post-construction verification procedures.
  • 3.1 Data Stage: LACap-50k Construction: Figure 2 organizes captions across seven audio-analysis levels and shows clip-duration, acoustic-attribute, sound-event, and caption-length distributions.The taxonomy covers signal, perception, semantic, cultural, pragmatic, musicological, and Foley event-sound facets.
  • 3.1 Data Stage: LACap-50k Construction: Post-hoc ASR auditing yields 6.23% micro-CER and 9.21% micro-WER across 36,809 transcript spans from 10,975 speech-heavy captions.The audit uses Qwen3-ASR-1.7B as the reference model; English captions are reported as cleaner than non-English captions.
  • 3.1 Data Stage: LACap-50k Construction: Non-speech event verification corroborates 3,770 of 3,985 eligible claims, while 151 are not corroborated and 64 remain uncertain.The evaluation uses OpenFLAM on 1,000 final captions and reports 94.60% corroboration, 3.79% non-corroboration, and 1.61% uncertainty.
  • 3.2 Training Stage: Layer-Curvature SFT (LC-SFT): LC-SFT combines caption-only MCQ rewards for on-policy candidate selection with layer-curvature reliability weights for token-level gradient allocation.Stage 1 provides paragraph-level capacity through standard SFT, while Stage 2 refines the resulting policy using its own sampled rollouts.
  • 3.3 Inference Stage: SCC-Verifier: SCC-Verifier generates multiple caption rollouts, converts each caption’s atomic claims into audio-grounded MCQs, and selects the candidate with the highest surviving-claim fraction.The verifier answers questions in audio-only mode with the candidate caption withheld, making selection depend on audio-grounded self-answering.

4 Experiments

The experiments evaluate caption-as-evidence QA, direct caption fidelity, open-ended captioning, judge robustness, and SCC ablations. LC-SFT and SCC improve performance across these evaluation settings, with the strongest gains on Omni-Cloze and additional benefits from data and inference stages.

  • 4.1 Caption-as-Evidence QA on Audio Benchmarks: Caption-as-evidence QA uses a frozen text-only judge to answer MMAU, MMAR, and MMSU questions from generated captions alone.This protocol measures how well captions preserve fine-grained audio information needed for downstream reasoning.
  • 4.1 Caption-as-Evidence QA on Audio Benchmarks: 68.3, 57.0, and 62.4 are LC-SFT’s scores on MMAU, MMAR, and MMSU, improving over the vanilla backbone by +3.1, +5.2, and +1.8.LC-SFT attains the highest score among open-source captioners on all three benchmarks.
  • 4.2 Detailed Captioning Evaluation on Omni-Cloze: 58.9% on Omni-Cloze is LC-SFT’s audio-visual score, exceeding the vanilla backbone by +44.8 points and Omni-Captioner-7B by +4.4.It surpasses Gemini 2.5 Pro by 14.8 points and trails Gemini3.1-Pro by 0.6.
  • 4.2 Detailed Captioning Evaluation on Omni-Cloze: Adding audio-only self-check to an audio-visual draft produces the highest Correct rate, lowest Missing rate, and best Net score, while Wrong rises from 2.99% to 4.66%.These outcomes are reported on the human-validated Omni-Cloze evaluation set.
  • 4.3 Analysis: LC-SFT improves all five open-ended Clotho metrics over SFT, including B-CLAP from 0.460 to 0.492 and FENSE from 0.401 to 0.433.The control uses audio-only free generation without questions, a caption-only judge, SCC-Verifier, or MCQ reranking.
  • 4.4 Ablation of the SCC Framework: 58.59 with curvature-only reweighting and 58.92 with the full objective compare with 57.10 for SFT, making token-level curvature the dominant LC-SFT contributor.The ablation reports +1.49 for curvature alone and +1.82 for both signals combined.

5 Conclusion

Self-Check Captioning (SCC) uses audio-grounded question answering to verify long-paragraph detailed audio captions across data construction, training, and inference. Its artifacts achieve state-of-the-art open-source results across multiple benchmarks and improve Gemini3.1-Pro by 8.6 Omni-Cloze points.

  • SCC verifies long-paragraph detailed audio captions with audio-grounded question answering throughout data construction, training, and inference.
  • LACap-50k, LC-SFT, and SCC-Verifier attain state-of-the-art open-source results on MMAU, MMAR, MMSU, and Omni-Cloze.
  • 8.6 Omni-Cloze points: applying SCC at the data and inference stages improves Gemini3.1-Pro, demonstrating model-agnostic utility.

Limitations

LACap-50k is intended for posttraining rather than pre-training because producing a comparably high-quality corpus at pre-training scale remains prohibitively expensive. The construction pipeline also spans seven audio-analysis levels and uses independent verification passes on complete audio tracks.

  • Limitations: LACap-50k’s 50,222 verified clips support supervised fine-tuning and on-policy refinement, but comparable pre-training-scale data remains prohibitively expensive.
  • Construction boundary: The seven-level taxonomy covers signal, perception, semantic, cultural, pragmatic or intent or stance, musicological, and Foley or event-sound facets.Each level expands into three subcategories used to formulate audio-only relisten questions.
  • Construction boundary: The released captions are checked against the taxonomy through facet-conditional vocabulary clouds spanning acoustic attributes and other captioned audio content.
  • Construction boundary: The construction pipeline uses complete original clips for drafting and complete extracted audio tracks for subsequent independent verification passes.Passes use 16-kHz mono WAV audio and do not propagate contextual state.

A.4 Transcript-Fidelity Metric Definitions

The transcript-fidelity audit extracts explicit transcript spans from final captions, compares them with Qwen3-ASR references using character- and word-level edit rates, and separately evaluates non-speech event claims.

  • Metric definitions: Transcript spans are extracted from final captions and compared with Qwen3-ASR references using per-span character error rate and word error rate.CER and WER use Levenshtein edit distance normalized by reference length at the corresponding level.
  • Metric definitions: Micro aggregation normalizes pooled edit counts and weights spans by reference length, whereas macro aggregation averages per-span rates equally.The same distinction applies to character- and word-level metrics.
  • Metric definitions: The audit also reports the share of spans with per-span CER at or below the 0.2 quality threshold.
  • Non-speech event audit: For non-speech event grounding, 3,985 eligible claims yielded 3,770 corroborated, 151 not corroborated, and 64 uncertain claims.These correspond to 94.60%, 3.79%, and 1.61%, respectively.

C.4 Relation to the Training Reward

The training reward and SCC-Verifier share audio-grounded multiple-choice question answering but differ in question source and verifier identity because they operate at different lifecycle stages.

  • MCQ source: Training MCQs derive from the data-stage SCC-verified gold caption, whereas inference MCQs derive from each candidate caption because no gold caption is available.
  • Verifier identity: Training uses a frozen external text model to score retention of gold-verified caption facts, while inference uses the trained model to audio-check its own candidate claims.
  • Framework role: Together with modality-subtraction self-checking during data construction, the training reward and inference verifier cover SCC’s three lifecycle stages.

D Omni-Cloze Audio-Only Input Results

The evaluation additionally measures Omni-Cloze audio-subset accuracy with audio alone, excluding video to isolate audio-understanding capability.

  • Audio-only evaluation removes video access from the Omni-Cloze audio subset.This setting complements the audio-visual evaluation, where both modalities are provided.

E LC-SFT Component Ablation

The LC-SFT ablation isolates response-level and token-level reweighting on top of Stage-1 SFT, while Omni-Cloze also reports an audio-only comparison.

  • LC-SFT component ablation: The 2×2 ablation evaluates SFT, response-only +Adv, token-only +Tok, and full LC-SFT variants.All variants are evaluated on Omni-Cloze’s audio subset with audio-visual input.
  • LC-SFT component ablation: +Adv applies only response-level advantage reweighting, whereas +Tok applies only token-level layer-curvature reweighting.Each component is isolated by holding the other reweighting factor equal to 1.
  • Audio-only input results: Table 9 evaluates models on Omni-Cloze’s audio subset under audio-only input, with the final row reporting the LC-SFT-trained model.Each model receives only the audio stream and no video.

F Relation between SCC-Verifier and the Training Reward

SCC-Verifier and the training reward share audio-grounded MCQ answering but serve complementary roles at training and inference.

  • Training-time reward: The training reward uses MCQs derived from an SCC-verified gold caption to measure verified audio-fact retention.A frozen external text-only language model scores candidate captions without audio access.
  • Inference-time verification: At inference, MCQs are derived from each candidate caption and answered by the LC-SFT model in audio-only mode.The candidate text is withheld, making the model assess whether its assertions are grounded in the audio.
  • Inference-time verification: Withholding candidate text prevents inference verification from collapsing into textual self-agreement.The check therefore tests correspondence between candidate assertions and the actual audio.
  • Complementary roles: The training reward teaches audio-grounded captioning from verified gold captions, while SCC-Verifier selects among samples without a gold reference.These mechanisms are complementary because they operate at different lifecycle stages.

G Experiment Setup

Experiments use Qwen2.5-Omni-7B with two-stage training, synchronized multimodal processing, multiple captioning baselines, and controlled evaluation protocols.

  • Training framework: Qwen2.5-Omni-7B is the captioner backbone for both training stages.Training uses Swift, DeepSpeed ZeRO-3, eight GPUs, bf16 mixed precision, gradient checkpointing, and seed 42.
  • Two-stage training: Stage 1 uses 44k LACap-50k clips for standard SFT, while Stage 2 uses a disjoint 1k clips for LC-SFT refinement.The stages differ in both training objective and data subset.
  • Multimodal input processing: Each clip is processed as synchronized video and audio, with the audio output head disabled so the model produces text only.Video frames are sampled at 0.2 frames per second, with 2–32 frames per clip.
  • Hyperparameters: LC-SFT uses K = 4 training responses, κ=2, and the last m=3 Thinker layers; inference uses M = 4 candidates at T =0.8.Inference also samples N verification MCQs per candidate, without fixing N.
  • Baselines: Baselines span proprietary and open-source audio-visual captioners evaluated on MMAU, MMAR, MMSU, and Omni-Cloze.Open-source comparisons include video-SALMONN, videoLLaMA 2, video-SALMONN 2, and Omni-Captioner-7B.
  • Evaluation protocols: Open-ended Clotho evaluation generates one audio-only caption per clip across the complete 1,045-clip split without question-based reranking.RefAlign-F1 measures soft alignment to five references, while audio grounding is evaluated separately.
Loading 2608.30713v1…